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INTRODUCTION 



Since 1990, state policymakers have been increasingly interested in the educational quality, 
productivity, and effectiveness of public colleges and universities. This move towards greater 
accountability has been precipitated in most states by decreased state resources for higher 
education, rising costs, and growing demands for access. Policymakers look to state-level accountability 
measures to guide state planning and budgetary decisions and to monitor the "public investment" in 
higher education. 

For accountability purposes, a growing number of states are adopting "performance indicators," or 
common measures for colleges and universities to assess and report their performance. Selected 
indicators range from easily quantifiable statistics such as faculty/student ratios to more qualitative 
measures such as plans to increase minority student enrollment. In most cases, indicators are published 
in a higher education annual report or "report card," which allows for institutional comparisons and 
provides a readily understandable format for a broad audience. 

Ten states have been among the leaders in developing new accountability policies and performance 
indicators: Colorado, Florida, Illinois, Kentucky, New York, South Carolina, Tennessee, Texas, Virginia, 
and Wisconsin. Their experiences form the foundation for this report. Collectively, the 10 states 
constiuite an interesting mosaic, presenting both differences and similarities in the manner in which each 
accountability process originated, how performance measures or indicators were developed, and for 
what purposes the results have been used. 

Peter T. Ewell and Dennis P. Jones set the context for understanding the rise of new accountability 
policies and performance indicators in their essay, "Pointing the Way: Indicators as Policy Tools in 
Higher Education," which comprises the first part of this report. In their essay, they define performance 
indicators, explain the reasons why states are interested in them, and offer some insights into the 
rationale behind them. Their discussion provides an introduction to the 10 state case studies that follow. 

In each of the 10 states, new accountability policies originated either with the legislature or the state 
higher education agency. In some states, the impetus came from legislative or gubernatorial efforts to 
restructure government services in general. In others, the review of higher education's performance was 
part of a general reform movement involving K-12 education. Where accountability policies were 
initiated by the governing or coordinating board, they were often designed to forestall a legislative 
mandate or were part of a broader set of policy initiatives perceived to be an important condition to 
maintaining of increasing institutional autonomy. 
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Despite diverse origins, nearly all the case-study states can trace the roots of the new accountability 
policies back to earlier assessment initiatives. State efforts to promote assessment of student learning 
began as early as the late 1970s, with most states joining the movement by the late 1980s. The 
difference between earlier assessment efforts and the more recent accountability policies, however, lies 
in their purposes and their processes. Earlier assessment efforts were decentralized and institution-based. 
Colleges and universities were encouraged to develop evaluation measures unique to their missions, 
permitting measurement over time but not interinstitutional comparison. This process is still ongoing in 
most states, where it has been useful in promoting student learning assessment and improvement in 
academic programs. 

New accountability policies, on the other hand, reflect the view that higher education needs to be more 
responsive to state concerns and more publicly accountable to a broader constituency that includes 
students, employers, parents, and the general public. While accountability requirements have built upon 
rather than replaced earlier assessment efforts, the added element of public reporting on a set of 
performance indicators gives those with a stake in higher education a better sense of what is being 
achieved with public resources. 

Responsibility for identifying the set of performance indicators varies among the states. Some originate 
with governing or coordinating boards, others in state legislatures, still others from the work of specially 
appointed commissions or committees. Most states share a common core of indicators, although the 
actual number of indicators typically ranges between 15 and 25 elements. This consistency has more to 
do with the ready availability of certain data than with broad consensus about what js most important. 
From state to state, efforts have been made to use these data only to compare institutions with peers and 
to be sensitive to circumstances that affect how data should be interpreted. 

How performance indicators are linked to planning and budget also varies by state. In some instances, 
the higher education agency was able to tie indicators directly to statewide or system planning efforts. In 
others, indicators were developed rapidly and thus connections are indirect at best. While only 
Tennessee has a substantial history of linking performance to funding, five other states have proposed or 
initiated such a program. Connections to the budget are more indirect in other states as higher education 
agencies attempt to "strike a deal" with state legislatures, trading greater accountability results for more 
flexibility in the use of resources. 

It is important to recognize that the case studies were written during the course of a year and are 
"snapshots" in time. The development of performance indicators has continued to evolve in each of the 
states. In some cases, the latest related rie^elopments are not reported. For example, Colorado 
lawmakers passed a 1994 bill that ties performance to incentive funding for higher education, Virginia 
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has taken additional steps towards implementing performance indicators, and Florida recently released a 
statewide accountability plan. Some of the more recent changes are referenced in the two analyses by 
Richard C Richardson, Jr., and Peter Ewell that follow the 10 case studies. 

Richardson's and EwelTs essays bring closure to this report by providing analyses and 
interpretations of performance indicator development and use in the 10 states. Richardson employs two 
classification models: (1) an input/output/outcome model; and (2) a quality definition model. His first 
model adapts indicator categories outlined in Reinventing Government: How the Entrepreneurial Spirit 
is Transforming the Public Sector. By definition, inputs are baseline measures of instructional inputs and 
monetary resources (e.g., student/faculty ratios, state appropriations per capita, and average class size). 
Outputs are measures of institutional production (e.g., course completion rates, total degrees awarded, 
and time-to-degree). Outcomes measure qualitative benefits to students and the state while taking into 
account institutional missions (e.g., student performance on nationally normed tests, placement of 
graduates, and results of alumni satisfaction surveys). Using this scheme, Richardson observes that 
states are currently far more interested in output indicators than inputs and show signs of a growing 
interest in outcomes. 

In Richardson's second model, performance indicators are assigned to five categories of quality: 
(1) transcendent quality, through institutional rank or reputation; (2) cost/benefit quality, determined by 
comparing institutional inputs and outputs with system and institutional goals; (3) process-based quality, 
assessed by examining output and outcome indicators with respect to explicit standards; 
(4) product-based quality as determined by the measurable attributes of graduates; and (5) user-based 
quality, measured by client satisfaction. Richardson concludes that each definition suggests ways of 
measuring quality by what is valued by various higher education stakeholders (e g., the state legis- 
lator, the student, the higher education administrator, and the general public). 

In his analysis, Ewell lists performance indicators according to categories conforming generally to state 
goals or domains: (1) instructional quality (inputs, processes, outcomes); (2) general efficiency and 
productivity; (3) condition of the asset; (4) diversity, access, and equity; (5) articulation and K-12 
linkages; and (6) relation to state needs. In the case-study states, policymakers emphasize instructional 
quality and show a growing interest in measuring efficiency and productivity. Ewell concludes that 
regardless of which indicators are chosen, they "must be seen as a tool to shape the future, not a 
collection of statistics to report the past." They must meet both the short-term needs for accountability 
and the long-term needs for planning and priority setting. 

Readers are invited to review the 10 case studies and to reach their own conclusions. Although 
performance indicator development and use are still preliminary in most states, the 10 states provide 
useful lessons for state policymakers and institutional leaders considering such an approach. 
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PROJECT AND REPORT BACKGROUND 



This report is one of two publications resulting from a two-year ECS project to examine state-level 
performance indicators. It is supported by grants from the U.S. Department of Education's Fund for the 
Improvement of Postsecondary Education (FIPSE) and The Pew Charitable Trusts (through the ECS 
Project on State Policy and College Learning). The second publication, Charting Higher Education 
Accountability: The Roots and Realities of State-l<evel Performance Indicators, is a guide for state and 
higher education policymakers. 

ECS has a long history of involvement with state higher education assessment and accountability policy. 
Beginning in 1985, ECS published Transforming the State Role in Undergraduate Education, which, 
among other things, recommended that (1) a state's system of higher education should be monitored 
to examine the extent to which it meets state goals for undergraduate education, and (2) the system 
should report its results periodically. In 1986, in conjunction with the American Association for Higher 
Education and the State Higher Education Executive Officers, ECS conducted a national survey of state 
assessment initiatives.The survey was repeated in 1989 as part of a project, supported by FIPSE, to help 
further the national dialogue on higher education assessment. The project culminated with the 
publication, Assessing College Outcomes — What State Leaders Need to Know, a policy guide that 
answers questions about the difficulties and rewards involved in using assessment policy to benefit both 
students and the state. 

This current project builds directly on such past work. ECS began this project, again with FIPSE 
support, by focusing on educational quality and how it is perceived and assessed by states. Cross-state 
forums of educational leaders and policymakers were held to test ideas about new broader definitions 
and understandings of quality. These discussions led to some key observations: 

• Measurement and reporting should be clearly tied to a broad state policy framework for higher 
education. 

• Accountability reporting needs to be viewed less as a product and more as an iterative process. 

• State policymakers are interested not only in educational quality but in productivity, efficiency, 
equity, and effectiveness. 

As the project unfolded, 10 case-study states were selected to provide further insight into new policy 
approaches related to these points. Additional support for the work with 10 states was provided by The 
Pew Charitable Trusts through the ECS project State Policy and College Learning, an effort that 
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examines how state policy can promote significant improvements in undergraduate teaching and student 
learning. 

In conducting the case studies, ECS looked at materials such as master plans, accountability or 
assessment legislation, and state-level reports. In addition, the heads of state higher education 
coordinating or governing boards or their designees were interviewed and given an opportunity to 
review drafts of the studies. Although the case studies were reviewed by outside parties, responsibility 
for the contents of this document rests with ECS. 
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POINTING THE WAY: INDICATORS AS POLICY TOOLS 
IN HIGHER EDUCATION 



Peter T. Ewell and Dennis P Jones 

In recent years, the policy context for state higher education has become more complex and the need 
for coherent policy more urgent. Consistent with the expansion of knowledge and technology^ 
college and university programs have grown in number and become increasingly diverse. Workforce 
requirements have also changed: a generation ago, the skills needed for global competitiveness could be 
taught in secondary schools; today, most are taught at the postsecondary level. Funding processes, too, 
have changed. In the past, public investment levels in higher education kept pace with changing 
expectations, while today such resources are increasingly scarce. As a result, the challenge to manage 
the higher education enterprise effectively in a manner consistent with public purposes has never been 
greater. 

As with health care and other complex and costly public enterprises, state governments are turning to 
statistical indicators as a tool to help develop and monitor effective higher education policy. The purpose 
of this analysis is to help state education leaders establish an appropriate state-level indicator system for 
higher education. It seeks, first, to describe such systems and how they work; and, second, to provide a 
set of comprehensive, policy-level questions state leaders can ask when reviewing the soundness and 
utility of any particular set of higher education indicators. 



INDICATORS AS POLICY TOOLS IN HIGHER EDUCATION 
What is an Indicator? 

Indicators can best be described as policy-relevant statistics produced regularly to support overall policy 
planning and monitoring at the national, state, or system level. Among the most prominent U.S. 
examples are the Department of Labor's unemployment rate and the Department of Commerce's report 
on Gross Domestic Product (GDP), both of which are used to measure the nation's economic health. An 
international example is the infant mortality rate used by the U.S. Public Health Service as one gauge of 
the effectiveness of the national health care systems. In fields such as these, indicators are carefully 
selected to provide specific insight and evaluation, purposes that apply to indicators whether they 
measure the nation's economy or its system of higher education. 



For purposes of this analysis, then, an indicator is a concrete piece of information about a condition or 
result of public action that is regularly produced, publicly reported, and systematically used for planning, 
monitoring, or resource allocation at the state or system level. Such indicators are intended to be used 
together, not singly or out of context. 

In the context of higher education, indicators are used to provide an indirect overview, often through the 
use of proxy measures designed to reflect trends and conditions accurately and effectively. They include 
statistics both about current practices (e.g., the proportion of freshman classes taught by full time 
faculty) and about key features of the higher education enterprise (e.g., the proportion of undergraduate 
curricula that require a thesis or "capstone experience" to obtain a degree). 

Motivations Behind Indicators 

Recently, a number of forces have combined to stimulate state-level interest in and justify the use of 
higher education indicators: 

Increasing complexity and size of the higher education enterprise as a whole 

Since 1965, enrollments in higher education have more than doubled, new kinds of institutions have 
been created, and the student bodies have become more diverse. Used properly, indicators can help make 
sense of this complexity by summarizing efficiently the current condition of higher education and the 
degree to which state objectives for higher education are being attained. 

Rising costs in delivering higher education and an eroding state resource base 

State budgets have become pressed at the same time that budgetary requests from public colleges and 
universities have grown. Well-constructed indicators potentially can provide an efficient and relatively 
unambiguous set of tools to help guide available state-level resources toward the problems and areas 
where they are most needed. 

Growing concerns about improving the linkages between public colleges and universities and the 
wider society 

Partly spurred by hard times, state policymakers have become concerned about the "return on 
investment" represented by public higher education. A well-conceived indicator system can reflect broad 
changes in societal need and effectively monitor higher education's contributions toward meeting 
economic and social needs. 



Particular concerns about revitalizing undergraduate education in state higher education systems 

Through a variety of assessment and improvement programs, many states have urged institutions to 
engage in ongoing improvement of undergraduate instruction. Used appropriately, indicator systems can 
help summarize the results of institutional assessment efforts and can both stimulate and monitor the 
implementation of recognized "good practices" in undergraduate education. 

To realize these potential benefits, however, policymakers must be aware that the use of indicators 
implies a particular model of how state policy is constructed. An approach using indicators first assumes 
the legitimacy of proactive, state-level intentions and actions with respect to higher education that go 
beyond the purposes of a single institution. While indicator systems can and have been constructed in 
the form of "report cards" reflecting the condition and performance of individual institutions, the 
optimal use of statewide indicators is systemic — that is, designed to guide the development of 
state-level policy as a whole. 

Indicator systems are also meant to Id long-term. Relying upon indicators implies an incremental policy 
approach that stresses continuous improvement rather than quick fixes. This, in turn, implies a rational 
approach to policy and resource allocation, characterized by the conscious use of concrete information in 
deciding what can and should be done. An important distinction needs to be made here, however. In 
using indicators, considerable policy attention must be directed toward determining how a given state's 
system of higher education functions as a whole and how its individual pieces interact; this differs from 
using statistics narrowly in an attempt to micromanage individual units, colleges, or universities. 

Strengths and Weaknesses of Indicators 

There are particular advantages and disadvantages to using indicators as policy tools: 
Strengths 

• They can enhance the process of state- and institution-level goal development. By making intended 
outcomes more visible and concrete, both educational leaders and the public become more aware of 
what needs to be accomplished. Explicitly tracking degree-completion rates for minority students, 
for instance, sends a far more concrete signal about what needs to happen than does the more 
general goal of increasing access. 

• Tfiey can help mobilize concerted action within the higher education community, as well as support 
for higher education awang the public at large. Concrete indicators provide a better focus for 
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applied policy than do more general directives. Reporting data on lower-division instructional loads 
and expenditures, for example, creates greater incentives to address directly the problem of lack of 
resources in the early years of undergraduate study than do more general exhortations to improve 
quality. 

• They can support and reinforce development of a rational set of state policies and institutional 
actions directed toward ongoing improvement Statewide indicators help document statewide needs 
and higher education's current progress in meeting them. Comprehensive data about the numbers 
and proportions of entering students deficient in basic algebra skills, for instance, help make the case 
for greater investment in remediation and show explicitly the levels, kinds, and locations of 
resources that will be needed across the state. Similarly, concrete data about state adult literacy rates 
call attention to the need for higher education institutions to work with other state agencies to 
address a common work force problem. 

Weaknesses 

But indicator systems also have drawbacks as policy levers: 

• They seldom tell policymakers and the public directly what they want to know. By their very nature, 
practical indicator systems tend to be indirect. Using indicators, policymakers can determine such 
things as which institutions are investing heavily in undergraduate study, what kinds of resources 
they are investing, and what many of the outcomes of this investment may be. What they will not 
have is a reliable single index of higher education quality. 

• They tend to create false short-term incentives for action. If the stakes associated with poor 
performance are high, institutions may act to maximize the numeric values of indicators without 
really changing practice or performance. Placing excessive emphasis on achievement test scores, for 
example, can strongly induce institutions to admit only the best students in the first place or to 
elimiir r«» weak students before they are tested. 

• They can focus attention on information-gathering itself, rather than on real action to change 
conditions. Because indicators are often technically complex and the changes they point to difficult 
to implement, excessive institutional effort may be expended on measurement issues unrelated to 
local improvement. Initiating common statewide basic skills testing, for example, may require 
significant new institutional investment without eliminating the need for institutions to maintain 
their own duplicative, local assessment systems. 
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Like any other policy tool, indicator systems are no panacea. In particular, their appropriate use demands 
careful planning and significant attention to how the resulting information fits within overall state-policy 
directions and what kinds of incentives and disincentives they create for institutional action. 



WHICH INDICATORS ARE NEEDED FOR HIGHER EDUCATION POLICY? 

Many indicators now proposed or in current use at the state level rely heavily on available statistics that 
are not necessarily the most informative. State higher education institutions and agencies, for example, 
routinely produce large volumes of statistics about their operations for accountability purposes. Among 
the most prominent are activity-level statistics such as enrollments and costs. What most legislators and 
the public want, however, are output or return-on-investment statistics such as degrees granted or 
retention and completion rates. To avoid creating false incentives for action that do not serve state 
purposes, policymakers must recognize that appropriate indicator systems must be multiple, 
comprehensive, and mutually reinforcing. 

Appropriate indicator systems can be constructed in many ways, but the best are guided by an explicit 
model of the assets contained in a state's higher education system and of the functioning of the system 
itself. At the same time, appropriate indicator systems generally contain data of two kinds: the current 
condition of higher education as a public investment (e.g., data on the age and deferred maintenance of 
facilities and capital equipment), and the degree to which particular policy objectives are being achieved 
(e.g., the rate of minority degree attainment in science and engineering). The first of these corresponds 
roughly to a business financial statement showing assets and liabilities for the enterprise as a whole, 
while the second resembles a business operations statement showing profits and losses for a given 
period. The contents of the first set of indicators will be relatively standard, while those of the second 
will depend primarily on the specific policy objectives that the state is seeking to accomplish. Both are 
important for sound decision making. 



Approaches to Developing Useful Indicator Systems 

These broad parameters, however, can encompass many distinct approaches to developing a useful 
statewide indicator system. Four approaches are commonly used. 
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Jnputs/Processes/Outcornes 



This approach is probably the most familiar and is often used in accountability reporting for K-12 
education. Its underlying model is a production process; the primary output to be measured is "value 
added/* Three types of indicators are most typically included: 

♦ Entering student numbers, characteristics, and ability levels 

♦ Per-student instructional expenditures, educational levels and backgrounds of instructors, types of 
instruction provided (e.g., curriculum design), actual instructional experiences (e.g., class sizes, 
types of requirements completed, amount of faculty contact), and student behavior (e.g., retention, 
choice of program, and courses, etc.) 

♦ Exiting-student numbers, characteristics, and ability levels (and the difference between each of these 
and the corresponding values for entering students) 

Resource Efficiency and Effectiveness 

As noted, higher education financial data are commonly collected and reported at the state level. For the 
most part, however, fiscal information has been used to ensure that appropriated dollars are spent legally 
and as intended. In indicator form, however, resource information can have other important uses. The 
most common is to monitor efficiency — particularly in the way physical resources such as faculty, 
space, and equipment are utilized and deployed. 

Less common but equally important uses are to monitor the current capacity and condition of critical 
institutional assets. For instance, have visible gains in output been purchased at the expense of potential 
long-term effectiveness in the form of deferred maintenance of physical plants or dangerous erosions in 
faculty compensation in key disciplines? The more prominent types of indicators under this model are 
as follows: 

♦ Current type, condition, and distribution (across institutions and programs) of key physical assets 
such as faculty, stafi, space, or equipment, and historical patterns in the above 

♦ Patterns in the actual utilization of both fiscal and key physical resources over time — for instance, 
facilities utilization by type of space, faculty teaching loads and research assignments, or 
instructional expenditures by discipline and by level 
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State Need and Return on Investment 



Increasingly, both policymakers and the public at large view expenditures on higher education as a 
strategic investment for the state as a whole, directly related to economic and manpower development 
and a more publicly aware and socially functional citizenry. This view of higher education suggests a 
wider set of indicators than those traditionally used. Some of the specific types of indicators included in 
this model are as follows: 

• Current and future work force needs, including types of positions and specific types of work force 
skills (e.g., oral communications skills, problem-solving skills, or interpersonal skills) required to 
meet future needs 

• The match between identified work force needs and higher education's current capacity to meet 
them (as shown in existing program capacities and locations, research and instructional emphases, 
and levels and types of degrees produced) 

• Overall changes in work force and citizen capacities for the state as a whole — for example, in 
levels of education completed or in assessed general skill levels 

"Customer" Need and Return on Investment 

Extending the notion of return on investment, higher education in any state has a wide range of 
customers including individual citizens, employers, and the public at large. Increasingly, both state and 
federal policies emphasize that these parties have a u right to know" about the past performance of 
colleges and universities. The underlying model here is one of informing consumer choice. The primary 
focus is determining the degree to which individual needs are met. Specific types of indicators are 
suggested by this approach: 

• Historical rates of persistence, degree-completion, and employment by field of study or type of 
preparation for entering students of different types 

• Job and career mobility of degree recipients by field of study or type of preparation 

• Proportion of employees requiring additional education or training to address particular identified 
skills deficiencies 
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CHOOSING AMONG THE MODELS 



These models are not mutually exclusive. Each provides a different way of looking at the relationship 
between higher education and its intended purposes. Most, in fact, contain elements either directly 
included or implied by one or more of the others. The point for policymakers is less to choose among 
them as much as it is to ensure that those responsible for developing any planned statewide indicator 
system recognize the need to be guided by an explicit policy framework of this kind. 

Regardless of the particular model chosen, effective indicator systems are best developed for a statewide 
system of higher education as a whole. The primary emphasis should not be on institutional policing 
through production of a report card, but rather on determining higher education's strengths and 
weaknesses and identifying how its overall performance might be improved. Effective indicator systems 
should be constructed to embrace multiple perspectives, including those of individual institutions, 
students, and such external constituents as employers and citizens. Appropriate policy requires prior 
consensus both about the goals to be sought and about how their achievement will be recognized by the 
various interested parties. Often, therefore, the same basic information can and should be communicated 
in indicator form to different audiences for different purposes. An excellent example is provided by the 
transfer of two-year college students to four- year colleges and universities. Here, appropriate statistics 
on transfer might be compiled in at least three ways: 

• From the perspective of the individual two-year college — for example, the proportion of 
degree-seeking students from a two-year college who successfully enroll in baccalaureate programs 
at a four- year institution 

• From the perspective of the state's system of higher education — for example, the proportion of 
baccalaureate degrees granted by public institutions to students who began their studies at a 
two-year college 

• From the perspective of the individual student — for example, based on past experience, the 
statistical chances of a student in a particular city drawn from a particular ethnic group completing a 
baccalaureate degree within six years after prior enrollment in a particular two-year college 

None of these potential indicators is inherently right or wrong. The choice of which to include depends 
entirely on the policy question being asked and from whose perspective. 
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SOME QUESTIONS THAT POLICYMAKERS NEED TO ASK 



Given the above context, policymakers should consider a number of specific criteria as they assess the 
utility of particular higher education indicator systems. An approach may look good on one dimension, 
possess severe handicaps in another, yet still be very useful for policy purposes. If it fails to meet too 
many criteria, however, its use may create substantial future difficulties. In using particular indicators, 
therefore, policymakers should recognize each system's specific strengths and weaknesses, and they 
must resist the temptation to view every statistic as equally valid and useful for all purposes. Among the 
most important questions that policymakers should ask about higher education indicators are the 
following seven: 

1. Does the indicator provide policy leverage for action to correct deficiencies? 

Some indicators merely offer information about the present state or condition of higher education 
without yielding much information about how to improve things; others provide direct guidance about 
what might be changed. A comprehensive statewide assessment of college-level communications skills, 
for example, may reliably indicate current levels of proficiency in written communication among college 
students, but it may also yield little information about what is required to improve performance. In 
contrast, while a statistic indicating the actual amount of writing required of college students may not 
show proficiency directly, it can provide much clearer direction regarding possible corrective actions. 

2. To what extent is the indicator susceptible to manipulation without real changes in the 
things that it is trying to measure or reflect? 

All indirect indicator systems are to some extent vulnerable to manipulation. As a result, multiple 
indicators should be designed to reinforce one another, and single indicators should be avoided in most 
cases, particularly if their values can be easily influenced. For example, statistics on the proportion of 
instructional expenditures dedicated to delivering lower-division course work depend greatly upon the 
ways individual courses are classified by institutions. If incentives to perform well on such indicators are 
compelling, institutions will quickly identify those modes of classification that yield maximum indicator 
values regardless of what they are actually doing. 

3. Is the indicator easily understandable to and credible for lay audiences? 

Some indicators are the result of complex statistical manipulations that say little directly to those 
unschooled in their interpretation. Others are "face valid'* for most observers — they not only measure 
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what they are supposed to measure, but they look like they do so and thus are able to communicate readily 
to lay audiences. Statistics on program completion rates or interinstitutional transfer rates, for example, are 
notoriously complicated when compared to more straightforward numerical information such as 
enrollments or numbers of faculty. This is because they depend upon numerous prior assumptions about 
who should really be classified as a degree-seeking student. In order for such statistics to be useful as 
indicators, their grounding assumptions and definitions need to be clearly visible to all who examine them. 

4. Does the indicatoi reflect the perspectives and concerns of multiple constituents? 

A good indicator system will contain several units of analysis, drawn from a range of client perspectives, 
including the student, the institution, and the system as a whole. Because consensus is required for 
meaningful action, the contents of any indicator system should potentially support the establishment of 
an informed dialogue among the appropriate parties. A statistic on the employment rates of program 
completers by field of study, for example, can be summarized from the student's point of view as the 
probability of being placed, from the college or university's point of view as a placement rate, and from 
an employer's point of view as the proportion of needed job areas filled each year. 

5. Against what benchmark will the indicator be compared in order to chart success or 
progress? 

Without a clear standard for comparison, the interpretation of any indicator is subject to considerable 
doubt. Indicators may be measured by direct comparisons among institutions, comparisons with 
identified peers, cross-state comparisons, or comparisons against established "best practices" or national 
norms. Indicators of faculty workload at a particular research university, for example, can be compared 
across a range of nationally identified peer institutions in order to establish its position with respect to 
industry standards for that type of institution. But the same statistics might be compared across all state 
institutions regardless of type as an indicator of their appropriate mission differentiation. 

6. To what extent is the indicator reliable and valid as a piece of data in itself, and how 
robust is it under typical conditions of missing or biased data? 

To be used in planning policy, indicators need to be founded upon valid and reliable measurement 
procedures, but they should also be designed to operate under less-than-ideal measurement conditions. 
Complex longitudinal data-reporting requirements, such as the proportion of entering students who 
complete a particular program of study within a defined time period, may be highly vulnerable to 
deficiencies and variability in institutional record-keeping procedures. 
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7. To what extent is the indicator practically obtainable at a reasonable cost? 

Many promising indicator systems fail simply because they are too expensive, too complex, too 
time-consuming, or too politically costly to implement. Often the simplest is the best even if it initially 
seems less technically attractive. Large-scale direct assessments of generic student abilities, as currently 
practiced in K-12 education for instance, are conceptually compelling as indicators, but they require 
heavy initial resource commitments for instrument development and may also require the expenditure of 
considerable political capital in order to overcome inevitable institutional resistance. Indirect 
indicators of curricular and teaching practice may be easier to collect and be just as useful for policy 
purposes. 



CONCLUSION 

In sum, indicators work best as tools of public policy when a wide range are developed and when they 
are clearly related to particular elements of system-level purpose. In using indicators, moreover, it is 
important to consider overall patterns and avoid focusing on small differences, which may result from 
unimportant local variations or simple chance. Most important, however, indicator systems should not be 
used to make final summative judgments. Instead, they should be employed to start discussions about 
concrete, systemic improvement. As industry has shown with the process of Total Quality Management, 
information is most valuable for improvement when it serves as a point of departure for improvement, 
rather than after the fact to determine whether any has taken place. The best indicator systems help 
students, institutions, and policymakers at all levels to recognize and take responsibility for their actions 
and accept the consequences. As experience in many public policy sectors has shown, it is only when 
responsibility of this kind develops naturally that real improvement occurs. 
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COLORADO 



Gordon B. Van de Water 



OVERVIEW 

Colorado was among the first case-study states to initiate higher education accountability as one 
of the major provisions of an omnibus higher education bill passed in 1985. As in other states, 
state-level assessment began as a decentralized effort. In consultation with public institutions, 
the coordinating board developed reporting guidelines and general assessment domains. Institutions 
were asked to develop specific assessment measures appropriate to their individual missions. After 
several years of institutional reporting, the coordinating board established the annual "Scorecard" to 
standardize measures of student and institutional performance. The Scorecard reports indicators by 
institutional sector (i.e., research universities, universities and colleges, and community colleges) and 
compares Colorado standings with those of other states. 

Colorado's assessment activities have resulted in institutional program improvement and better 
coordination among campus assessment administrators and coordinating board staff. The Scorecard has 
met a pressing need for greater accountability. Legislators likely will continue to push for assessment 
measures that are more easily quantifiable and allow for Institutional and programmatic comparison. 



CHRONOLOGY AND CONTEXT 
System Design and Evolution 

The Colorado Commission on Higher Education (CCHE), Colorado's statewide higher education 
coordinating board, was established in 1965, long after individual colleges and universities had 
developed their own identities and means of interacting with state government Initially, CCHE was a 
weak newcomer with little clout. Caught between institutional leaders looking for an advocate in the 
state capital and elected leaders concerned with limiting institutional aggrandizement, CCHE had 
difficulty establishing its niche in the public policy arena. With the passage of an omnibus higher 
education bill in 1985 (House Bill 1187), however, the role of CCHE was strengthened and higher 
education accountability requirements were established. 
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The higher education system for which CCHE has planning and policy responsibility consists of the 
multi-campus University of Colorado, governed by the publicly elected University of Colorado Board of 
Regents; the multi-campus Colorado State University system, governed by the State Board of 
Agriculture; the University of Northern Colorado and the Colorado School of Mines, each governed by 
its own board of trustees; 4 state colleges governed by the Trustees of the Consortium of State Colleges; 
11 state-sponsored community colleges under the control of the State Board of Community Colleges and 
Occupational Education; and 4 local district colleges. In 1992, these 28 institutions served more than 
168,000 Colorado residents and awarded 27,017 degrees. 



ASSESSMENT OF INSTITUTIONAL EFFECTIVENESS 

Since 1985, CCHE has been fully occupied with fulfilling the many responsibilities of HB 1187. The 
Higher Education Accountability portion of HB 1187 begins by declaring the intent of the general 
assembly in the following words: 

(a) That institutions of higher education be held accountable for demonstrable improvements in 
student knowledge, capacities, and skills between entrance and graduation; 

(b) That these demonstrable improvements be publicly announced and available; 

(c) That institutions express clearly to students their expectations of student performance; and 

(d) That these improvements be achieved efficiently through the effective use of student and 

institutional resources of time, effort, and money. 

The statute also specifically mandates that colleges and universities "design and implement a 
systematic program to assess the knowledge, capacities, and skills developed by students in academic 
and co-curricular programs/* According to CCHE, this legislation calls for both assessment and 
accountability: 

[while the legislation] was primarily designed to be "assessment" oriented, i.e., gathering 
information about the performance of students, it also established an expectation of 
"accountability" by requiring institutions to publicly disclose the results of student assessment. 1 
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These dual purposes of assessing students and holding institutions accountable has led to some 
confusion about the nature of the program and the purposes for which its results are used. 

As developed by CCHE in cooperation with the institutions of higher education, the process has five 
parts: 2 

1 . A CCHE assessment policy that requires each institution to assess general education, disciplines, 
retention/graduation, after-graduation success, and student/alumni satisfaction 

2. Institutionally developed goals for undergraduate education, a plan for assessment, and assessment 
measures for each of the above areas 

3. An annual reporting cycle, beginning in 1989, during which each institution reports its results to 
CCHE 

4. A CCHE review and comment process that provides constructive criticism to institutions and seeks 
common threads that might lead to statewide policy changes 

5. An annual report to the legislature by CCHE on progress toward the objectives outlined in HB 1 187 

Three years into the reporting cycle, a content analysis of the 112 institutional reports submitted to date 
reveals that colleges and universities use assessment for program improvement and not simply as an 
exercise to comply with a statutory mandate. Specific results include the following: 

• Faculty are increasingly involved in interpreting assessment data and developing changes based on it 

• Assessment results are fostering an increasing rate of change in courses and instructional methods as 
well as in support services 

• Assessment measures are evolving into a test of the application of knowledge rather than rote memory 

• Changes designed to improve teaching and learning are becoming more common as faculty become 
more comfortable with the tools of assessment 

• Institutions are meeting the assessment goals of the statewide coordinating board 

In 19^2, during the development of CCHE's new master plan, CCHE reviewed the higher education 
accountability program and concluded that it "appears to be satisfactorily achieving its objectives at this 
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point in time." 3 As CCHE noted, even institutions that generally resisted state-level mandates seemed to 
consider assessment activities valuable: 

Individual campuses use the results of assessment to make curriculum changes that improve 
instruction, and ultimately, student learning. The assessment process itself can be a useful tool to 
ensure dialogue about curriculum both within and between academic areas. . . . Assessment can 
also help to shift institutional focus from research to instruction. As another faculty member 
recently observed, What gets evaluated is what gets noticed. 4 



STATE-LEVEL INDICATORS 



HB 1187 mandates institutional participation in assessment and accountability but provides wide 
latitude in developing approaches suitable for each individual institution. From the beginning, 
institutions were expected to develop assessment programs addressed to each institution's stated 
objectives for undergraduate education. While the statute describes in general what colleges and 
universities are to assess, the only additional requirement is that "the instruments and methods used [be] 
appropriate. . . As a result* institutions have chosen a variety of assessment techniques to accomplish 
the state mandate, including the Major Field Assessment Test (MFAT), student portfolios, senior 
capstone projects or seminars, performance assessment, and a variety of surveys: 

• At the University of Colorado at Boulder, the 1990-91 accountability report provides 53 pages 
of detail on how 41 departments, programs, schools, and colleges carried out outcomes 
assessment plans. The report covers everything from an alumni satisfaction survey to the ability 
of kinesiology students to interpret scientific journal articles. 

• Colorado State University uses the ACT-COMP to establish a clear picture of their 
undergraduates. Five years of results indicate that CSU students score above the national norm 
in five categories and at the norm in one category. Results have been used to develop local 
assessment procedures. 

• At Metropolitan State College of Denver, after-graduation performance evaluations by employers 
show very high ratings for graduates in the areas of ability to learn, sense of responsibility, and 
initiative. Lower scores were reported for leadership skills, speaking, and writing. 
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While these institutional reports correspond to a CCHE list of criteria, thereby providing some 
continuity, one outcome of this approach has been "campus-specific" institutional reports that cannot be 
aggregated into state-level indicators. In order to provide the public with a standardized measurement of 
performance of students and institutions, CCHE worked with the governor's office, key legislators, and 
institutional leaders to develop the Scorecard on Colorado Public Higher Education: How the Public 
Higher Education System and Its Students Perform on Selected Measures, Organized around four 
systemwide educational values, the Scorecard started with 15 measures: 

Value 1: Educational Excellence 

Measures: 

• ACT and SAT test scores of first-time freshmen 

• Performance of Colorado graduates on graduate and professional school examinations 

• Average faculty salary 

Value 2: Educational Access and Diversity 
Measures: 

• Availability of academic programs 

• College participation rate 

• Graduation rates by ethnicity 

• Availability of student financial aid 

• Faculty diversity 

Value 3: Efficiency in the Delivery of Education 

Measures: 

• Student/faculty ratios 

• Sustained financial commitment to instruction 

Value 4: Adequate Resources for the Delivery of Education 

Measures: 

• Total revenues per student 

• Alumni and private contributions 

• State appropriations per resident student 

• State appropriations per capita 

• Grant and contract dollars per faculty full-time-equivalent (FTE) 

In addition to these measures, future reports will include items on student satisfaction, alumni 
satisfaction, employer satisfaction, academic performance of two-year transfer students, and 
administrative costs as a percent of total costs. 
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The final section of the Scorecard, "Recent Trends," provides contextual information on enrollments, 
degrees granted, appropriations, and tuition revenue. This information is helpful in interpreting the 
indicators. 



REVIEWING AND REPORTING OF RESULTS 



The Colorado Higher Education Accountability Program (HEAP), mandated by HB 1187, requires 
annual institutional reports to be made to CCHE regarding the status of implementation efforts; these 
reports are to be compiled by CCHE into a statewide student-outcomes assessment report. In complying 
with this requirement, both CCHE and the institutions encountered initial difficulties. 

The first round of reports (February 1990), for example, received poor grades from CCHE 
commissioners, primarily because they felt that institutional responses suggested that institutions were 
not taking the assessment mandate seriously. This led to a period of intense activity that served both to 
enlighten CCHE commissioners about the difficulties of launching assessment efforts and to strengthen 
working ties among CCHE staff, members of the CCHE Academic Council, and institutional assessment 
coordinators. These close working relationships proved beneficial in improving the reporting process in 
subsequent years. 

A second difficulty arose concerning the interpretation of institutional accountability. Because each 
institution had built its own assessment plan as it deemed appropriate, reports did not easily combine to 
create a set of state-level indicators. While seen as wholly appropriate when viewed through institutional 
eyes, it was a decided shortcoming to legislators interested in comparing institutional performance. 
While CCHE had consistently advocated the need for individual institutions to design and implement 
their own assessment programs, it also recognized the accountability interests voiced by state 
policymakers. To address these concerns, CCHE developed two responses. 

First, beginning with the third annual report, CCHE went beyond summarizing institutional 
assessment efforts to tease out issue areas with statewide policy implications. Three issues were 
identified: 

Teacher Education Programs — Because CCHE believed that "institutions have not developed 
meaningful assessment techniques for teacher education programs,*' 5 college of education deans weie 
designated as responsible parties to address this issue. Discussions are currently underway. 
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Effectiveness of Basic Skilis Instruction — Basic skills instruction surfaced as a result of 
information reported by three community colleges on student outcomes. The issue is being addressed by 
the Basic Skills Committee of the statewide community college board. 

Minority Retention/Graduation Rates — The issue of minority retention/graduation rates 
persists despite years of CCHE emphasis on affirmative action. CCHE plans to expand its efforts to 
share successful models from other institutions together with staff assistance with developing cultural 
diversity plans. 

CCHE's second response was to develop and publish the annual Scorecard referred to in the 
previous section. The Scorecard does not report on individual institutions. Instead, institutions are 
grouped into three sectors: research universities, universities/colleges, and community colleges. 
Although this approach does not permit individual institutional comparison, it provides a mechanism for 
reporting basic indicator information consistently over time without interfering with campus-based 
assessment practices. Initial experience shows that the Scorecard is not used frequently by the public, 
but it is used as a background source by the media and elected officials, although the latter would prefer 
to see data presented by individual institutions for direct-comparison purposes. 

Other kinds of assessment activities — institutional program review, accreditation, and CCHE special 
focus reports and policies — also produce assessment information, some of which is used in the 
Scorecard. Coordinating these assessment activities at various levels leads to an efficient system that 
serves the needs of both the campus and the capital. Some campuses use assessment findings in 
subsequent program reviews, while some connect assessment activities more closely to accreditation 
reviews. Assessment data are also beginning to be used in the planning process as leaders develop trust 
in the data. To date there have been no direct connections to the budget process. There is, however, 
interest in examining the range of assessment activities to determine where reporting requirements can 
be merged and/or reduced. 



CONNECTIONS TO OTHER PROCESSES 

HB 1 187 made one very explicit, punitive connection between assessment activities and money: 

Commencing July 1, 1990, the commission is authorized to retain a sum not in excess of 2 
percent of the appropriation for any institution which has not implemented or is failing or 
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refusing to some degree to implement any part of the higher education accountability program 
or fails to comply with the policies and standards of the commission. 



The former CCHE executive director felt this rule initially played a useful role in motivating 
institutions to take assessment and accountability efforts seriously. To date, CCHE has not invoked this 
"two percent rule." But, as educational analysts Francis Griffith and Stephanie Cunningham point out, 
the commissioners came close on one occasion: 

Commissioners were distressed by the small amount of data provided by the colleges and 
universities, especially the lack of data showing trends in student performance over time. This 
situation led them to believe that the schools were not taking the assessment mandate seriously. 
Unhappy Commissioners threatened io recommend that the Legislature withhold up to 2 percent 
of state funds from certain institutions. 6 

/ 

As previously noted, connections to program review and accreditation are now beginning to develop at 
the campus level. At the state level, Goal Six of the new Colorado Higher Education Master Plan calls 
for strengthening "higher education's accountability to the citizens of Colorado by continuing to 
implement meaningful programs of assessment" and beginning to "[integrate] assessment with other 
policies that monitor quality, streamline procedures, and reduce associated costs." As of this writing, 
these efforts are under discussion and do not yet include the budget process. 



IMPLEMENTATION PROBLEMS 



A top-down mandate is never warmly welcomed by those responsible for delivering any social 
service, and higher education is no exception. One early and predictable reaction from campus faculty 
was a search for hidden agendas on CCHE's and/or the legislature's part. Faculty suspected that 
assessment results might be used to eliminate academic programs. They were wary that initial 
institutional flexibility would soon give way to a standard assessment format imposed on all 28 public 
institutions, with the data used to punish a particular campus. They doubted the state's ability to use 
assessment results prudently; would the state, they wondered, resist the temptation to make 
inappropriate comparisons among programs or campuses? In addition, this new assessment program 
seemed similar to another CCHE accountability initiative called Program Review. Faculty were unsure 
whether the two processes were connected. 
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Initial fears eventually settled into substantive discussion regarding each institution's comprehensive 
assessment plan. In spring 1989, CCHE approved the plans submitted the previous fall. In October, the 
first annual implementation report came due, and in February 1990 CCHE's first annual statewide report 
was presented to the CCHE commissioners. 

The commissioners' negative response to the report led to earnest efforts to improve the reporting 
process and to create better working relationships between CCHE staff and campus assessment 
coordinators. By the second annual report, all players understood the rules of the game, and CCHE 
commissioners responded positively to the new reports. Subsequent discussions centered on further 
improvement to the process, refinement in the reporting requirements and cycles, and creation of 
positive incentives for institutions with a record of exemplary performance. 



CONCLUSION 



In Colorado the emphasis appears to be on the slow process of changing campus culture through the 
introduction of new forms of assessment and the use of results to change both student and institutional 
assessment. At CCHE, the view is that this change will take one to two decades but will fundamentally 
alter the nature and quality of the information used to assess performance. CCHE staff see this as a 
process that will lead to substantial improvement in both student and institutional performance. In this 
sense, they are deliberately "going slow to go fast." Although the measured pace and lack of 
comparability frustrates some legislators, results to date indicate that campus leaders find assessment to 
be increasingly useful. A future challenge will be protecting the decentralized approach that promises 
real long term results while still developing measures of accountability useful and informative for 
elected leaders. 



NOTES 

1. CCHE, Master Plan Background Paper: "How Can Postsecondary Education Be More Accountable to 
Colorado Citizens?" (Denver CCHE, September, 1992), p. 3. 

2. This section borrows heavily from Stephanie J. Cunningham's article, "Implementing an Accountability 
Statute," prepared for presentation at the AAHE Assessment and Continuous Quality Improvement Conference, 
June 9-12, 1993. 
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3. "How Can Postsecondary Education Be More Accountable to Colorado Citizens?" p. 11. 

4. "How Can Postsecondary Education Be More Accountable to Colorado Citizens?" p. 4. 

5. CCHE, "1992 Student Assessment Report" (Denver: CCHE, 1992), p. 12. 

6. Francis A. Griffith and Stephanie J. Cunningham, Ttiree Years Into Mandated Student Outcomes Assessment: 
Some Learnings at the Institutional and System Levels (Denver CCHE, 1992), p. 6. 
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FLORIDA 



Gordon B. Van de Water 



OVERVIEW 

Florida came early to the assessment of student learning by establishing the College Level 
Academic Skills Test (CLAST) in 1983. CLAST is a "rising junior" exam that tests basic verbal 
and mathematics competencies. Unlike other case-study states, Florida did not establish 
assessment in the mid-to-late 1980s for the primary purposes of institutional improvement. Rather, in 
1991, the legislature passed accountability legislation for the state university and community college 
systems. Legislative intent was to "monitor performance at the system level" in instruction, research, 
and public service. The statutes were explicit about the general areas to be measured: academic 
productivity, efficiency, quality, and equity. The Postsecondary Education Planning Commission (PEPC), 
an advisory group to the Florida State Board of Education, proposed general goals for statewide 
measures, including indicators that were to be related to specific state goals, contain key measures for 
each goal, allow for comparison with peer systems, and be reported to the public in an understandable 
format. 



CHRONOLOGY AND CONTEXT 



System Design and Evolution 

Prior to 1945, Florida had only 2 public two-year colleges and 3 public universities. In response to rapid 
population growth since World War II, however, Florida created 26 new community colleges and 6 
public universities (a seventh will be opened before the turn of the century). During this period, the 
emphasis clearly was on providing expanded educational opportunity through the provision of 
additional spaces for students. 

This rapid expansion led to basic alterations in the governance arrangements for higher 
education. Since 1985, the chief policymaking and governing body for public education in Florida has 
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been the Florida St^te Board of Education. 1 Its powers were amended in 1968 to reflect the growth of 
the state's colleges and universities. 

In 1965, the expanded public university sector came under the statutory jurisdiction of the newly created 
Board of Regents of the State University System, which was given responsibility for planning, 
institutional budget review, the systemwide legislative budget request, and program approval. In 1983, 
the State Board of Community Colleges was formed and charged with "establishing and developing 
rules and policies which will ensure the operation and maintenance of a state community college system 
in a coordinated, efficient, and effective manner." 2 

Advising the State Board of Education is the Postsecondary Education Planning Commission (PEPC), 
created in 1980 and charged with preparing a master plan for postsecondary education every five years. 
Other responsibilities include "recommending to the State Board of Education program contracts with 
independent institutions; advising the State Board regarding the need for and location of new programs 
and branch campuses of public postsecondary institutions; reviewing public postsecondary 
education budget requests for compliance with the State Master Plan. . . ." 3 

The State University System Board of Regents, the Community College Board, and PEPC are each 
required to prepare master plans intended to guide future development of postsecondary education in the 
state. Although educators expend much effort in this planning, the results have not been used 
consistently in the legislative process. According to a PEPC report, when it comes to legislative policy 
formulation, "relationships between legislators (and legislative staff) and higher education professionals 
have been characterized by more than the usual skepticism. Consequently, legislation tends to be more 
specific and constraining than in many other states." 4 

Student Demographics 

While similar in appropriations per full-time-equivalent (FTE) student, higher education in Florida 
differs from its counterparts in other similar-sized states by having fewer colleges and 
universities, lower participation rates, lower tuition levels, and much higher enrollment growth. 
Currently, Florida's 9 public universities enroll approximately 100,000 FTE students, and its 28 
community colleges approximately 200,000 FTE students. Most Floridians begin their college careers at 
a community college. Florida has no state colleges. Annually, state universities enroll approximately 12 
to 14 percent of the recent high school graduating class. 

Within this enrollment, African-American students are significantly underrepresented. Although 
approximately 20 percent of Florida's high school graduates are African-American, they represent less 
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than 14 percent of the full-time postsecondary enrollment and earned fewer than 7 percent of the degrees 
awarded in 1991-92. Hispanics, on the other hand, are enrolled much more closely in proportion to their 
presence in the overall population and achieve degrees at rates similar to those of the Caucasian 
population^ 

Chronology of Related Priorities and Initiatives 

The Florida State Legislature has been the primary initiator of assessment activities. The legislature took 
its first formal steps towards a mandated testing program in 1979 when it directed the State Board of 
Education to adopt minimum academic standards for college students in the areas of communication and 
computation. 

Acting on the 1979 directive, the board created the Articulation Coordinating Committee to oversee the 
standard-setting process. This committee created another entity called the Essential Academic Skills 
Project, charged with recommending standards generated through faculty and institutional involvement. 
Using results from more than 2,600 faculty and staff surveys, a statewide task force developed a list of 
essential skills that college students must attain. Following a round of review and comment, a final list 
of competencies was adopted by the statewide task force in December 1980. 

At the same time, the Standing Committee on Student Achievement reviewed commercially available 
testing materials and concluded that no tests existed that could adequately measure student 
achievement of the types of skill that the statewide task force had identified. This led to the formation 
of the College Level Academic Skills Project in 1981, designed to develop, implement, and maintain a 
sophomore-testing program. A revised list of competencies was approved by the board in September 
1981 and served as the basis for the development of the College Level Academic Skills Test (CLAST), 
the "rising junior" exam required of all college sophomores since 1983. 

CLAST passing scores were phased in during the period from 1984 to 1989. Over time, passing scores 
for reading were raised. The scale for evaluating writing skills scores was revised also, with the overall 
effect of raising the minimum needed to pass the essay portion of the test. The mathematics scores were 
only partially raised, however, after concerns were expressed by legislators, community college 
presidents, and other educators about potential adverse impacts on student completion rates. 

During 1988, external consultants reviewed CLAST and found it to be reasonably reliable and valid. 
They found no evidence of gender, ethnic, or racial bias. 5 At the same time, they suggested that th- test's 
degree of difficulty might be more appropriate as a measure of entry-level basic skills rather than those 
expected of college sophomores. PEPC described the study this way: 
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Since its inception the College-Level Academic Skills Test (CLAST) has generated a 
tremendous amount of interest and effort at both the state and institutional level on the part of 
students, faculty, administrators, and policy makers to place a renewed emphasis on the 

importance of general education The Commission continues to believe that performance 

will rise to meet expectations and that diminished standards do no service to those who may 
initially experience difficulty with the t^L 6 

Since CLAST is required for admission to the upper division, it is one means of assessing student 
performance prior to the start of upper division work. In addition, Florida has addressed issues of 
transfer and articulation as a way to enhance the flow of students through the system and increase 
institutional efficiency and effectiveness. 



SYSTEM AND STATE-LEVEL INDlCAf ORS 

As with other student assessment efforts, accountability measuies in Florida originated in the Florida 
legislature. The 1991 legislature passed two bills aimed at implementing an accountability process for 
both the state university and community college systems. In addition, proviso language in the 1991 
appropriations bill directed PEPC to "conduct a review of the status of outcomes assessment in Florida 
in comparison with model practices in place in other states, individual institutions and accrediting 
bodies." 7 The remainder of this case study describes these three legislative actions and resulting events. 



The State University Accountability Legislation 

Section 240.214 of Florida's education law expresses the legislature's intent as follows: 

It is the intent of the Legislature that an accountability process be implemented which provides 
for the systematic, ongoing evaluation of quality and effectiveness in the State University 
System. It is further the intent of the Legislature that this accountability process monitor 
performance at the system level in each of the major areas of instruction, research, and public 
service, while recognizing the differing missions of each of the state universities. The 
accountability process shall provide for the adoption of systemwide performance standards and 
performance goals for each standard identified through a collaborative effort involving the State 
University System, the Legislature, and the Governor's Office. 8 
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The legislatively mandated accountability process required the board of regents to prepare the following: 

1. An evaluation of the production of classroom contact hours to be completed by December 31, 
1991 and annually thereafter 

2. An accountability plan by October 1, 1991 

3. An annual accountability report, beginning December 1, 1992 

Finally, the act specified that beginning January L 1993, "the Board of Regents shall conduct an annual 
evaluation of the performance of the Chancellor and the state university presidents in achieving the 
performance goals established in the State University System accountability plan. . . 

The indicators required by the statute cluster around four issue areas: (1) academic productivity 
(including credit hours produced, degrees awarded, and contact hours of instruction); (2) efficiency 
(including length of time to degree, number of credits required for a degree, and classroom utilization); 
(3) quality (including pass rates on professional licensure examinations and ratings of alumni, parents, 
clients, and employers); and (4) equity (enrollment, progression, retention, and graduation rates by race, 
gender, and disability). In addition, the statute gives the following direction to the Office of the Auditor 
General to: 

conduct an assessment of the State University Accountability plan . . . which must include the 
following components: 

1. Evaluation of the extent to which the performance standards included in the plan are valid, 
reliable, and can be measured; 

2. Assessment of controls and procedures to be established . . .; 

3. Evaluation of the year, established by the State University System, which shall serve as the data 
base line for each performance standard; and 

4. Assessment of the basis of the weighted value formula, established by the State University System 
for each performance standard. 

The auditor general's report of December 31, 1991, generally approved of the system's approach to 
implementing the accountability statute but identified several data concerns and recognized the need to 
expand the assessment to the graduate, research, and service missions of the state's public universities. 9 
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The Community College Accountability Legislation 

Passed during the same session (1991), the community college legislation stipulated a somewhat 
different accountability plan: 

It is the intent of the Legislature that a management and accountability process be implemented 
which provides for the systematic, ongoing improvement and assessment of the improvement of 
the quality and efficiency of the State Community College System. 10 

This legislative mandate required the State Board of Community Colleges to (1) prepare an 
accountability plan by January 1, 1992; (2) implement the plan by December 31, 1994; and (3) submit 
annual reports beginning December 3i, 1992. 

In addition, beginning January 1, 1993, the executive director of the State Board for Community 
Colleges and the president of each college are to include in their annual evaluations a section on the 
achievement of the performance goals established in the community college accountability plan. For 
community colleges, institutional-effectiveness indicators required by the statute cluster around the same 
four issue areas as for the universities but specified differently. The indicators by issue area are 
(1) academic productivity (including graduation rates of associate degree-seeking su 'dents compared to 
first-time enrolled students), (2) efficiency (including job placement rates of vocational students and 
student progression by admission status and program), (3) quality (including student performance on 
CLAST examinations, grade-point average for transfer students, and performance on licensure 
examinations), and (4) equity (minority student enrollment and retention rates). Unlike the state 
university statute, the community college statute does not contain any requirement for a review by the 
Office of the Auditor General. 

The Postsecondary Education Planning Commission Study 

Through proviso language in the General Appropriations Act, the 1991 legislature directed PEPC "to 
review outcomes assessment in Florida higher education institutions." 11 The study revealed that Florida 
had no comprehensive state-level assessment policy, and that ongoing, systematic, and comprehensive 
assessment for the purpose of evaluating overall institutional effectiveness had not been a priority at 
most colleges and universities. Based on these findings, the commission reached several conclusions: 

1. Assessment of educational outcomes should be an integral and continuous part of the 
functioning of a postsecondary institution. 
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2. Consistency and comparability of assessment measures are important, and . . . should include . . . 
instruction, research, and public service. 

3. The educational process and student learning should be central to institutional assessment plans. 

4. The State needs to address fundamental and systemic issues related to the structure, financing, 
management, productivity, and delivery of postsecondary education. 

5. Conflicting forces internal to the institution, especially the public universities, have the potential 
of creating impediments to institutions' responsiveness to the needs of such diverse groups as 
government, business, industry, private organizations and associations. 

6. Concurrently, business and industry have a responsibility to determine specifically what their 
needs are and articulate these clearly to colleges and universities. 



These conclusions led to the recommendation that a statewide assessment process be delayed until more 
fundamental issues of management, structure, productivity, and financing could be resolved through the 
master planning process. In October 1993, PEPC completed the new Master Plan for Florida 
Postsecondary Education. Each issue was addressed in the plan, and it was reaffirmed that 
accountability was to remain the principal way of responding to productivity questions. In addition, 
assessment of state and institutional missions as well as accomplishment of goals was to be an integral 
part of postsecondary education. PEPC called on state-level governing and coordinating boards to 
provide guidelines and assistance to their institutions in designing and implementing a comprehensive 
outcomes-assessment program to examine institutional effectiveness. 

Florida is currently working toward a set of state-level indicators. At this time, the closest 
approximations are the system-level performance standards required by the 1991 statutes described 
above. For the State University System, the accountability plan "must include, at a minimum, data on 
the following performance standards": 

1. Total student credit hours produced, by institution and by discipline 

2. Total number of degrees awarded, by institution and by discipline 

3. Total number of contact hours of instruction produced, by faculty, by institution, rank, and course 
level 



4. Pass-rates on professional licensure examinations, by institution 
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5. Institutional quality as assessed by follow-up surveys of alumni* parents, clients, and employers 



6. Length of time and number of academic credits required to complete an academic degree, by 
institution and by degree 

7. Enrollment, progression, retention, and graduation rates by race, gender, and disability 

8. Student course demand analysis 

9. Classroom utilization 

The State University System's accountability plan, dated October 1, 1991, interprets the legislative 
language in the following words. The emphasis is to be "principally on undergraduate education. None 
of these measures is directed primarily towards graduate and professional education, to say nothing of 
research or service. . . . Thus, this report should be viewed as just the first phase in an evolutionary 
process that will eventually encompass the entire mission of our State University System." Perhaps 
because of limited time, the plan does not exceed the minimum requirements of the statute. Full 
implementation of the process was expected to be complete by December 31, 1993. 

For the community college system, the accountability plan was to address the following issues: 

1 . Graduation rates of AA and AS degree-seeking students compared to first-time enrolled students 
seeking the associate degree 

2. Minority student enrollment and retention rates 

3. Student performance* including student performance rates on college level academic skills tests, 
mean grade-point averages for community college AA transfer students, and community college 
student performance on state licensure examinations 

4. Job placement rates of community college vocational students 

5. Student progression by admission status and program 

6. Other measures as identified by the Postsecondary Education Planning Commission and approved 
by the State Board of Community Colleges 
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The Florida Community College System 1992 Accountability Report summarizes the work of the 
Accountability Implementation Committee during 1992. The two main outcomes of this effort were the 
establishment of a timeline for implementation and the development of "outcome measures that address 
the central issues adopted in the legislation." Like the state university system, the community college 
plan does not go beyond the minimum required by the legislation. The report notes that "numerous other 
processes exist and are under development to address the issues of accountability" and encourages 
diverse agencies to use common data definitions and reporting formats to meet multiple requirements as 
a way of using resources most efficiently. The report also comments that "no specific funding to support 
the development and implementation of this plan has been provided to the colleges or to the Division of 
Community Colleges." The state board had unsuccessfully requested funding in 1992-93 (the 1993-94 
request is for $2.73 million). While reaffirming commitment to accountability, PEPC concluded the 
report by noting the difficulty created by a lack of funds: 

If additional funding is not provided, the committee and the system are committed to developing 
alternative procedures and outcome measures that would address the issues in the accountability 
statutes. However, the outcome measures developed in this document are considered to be the most 
reliable, effective method of assessing the success of the colleges and students. 



NEXT STEPS 

Clearly, there is legislative interest in developing statewide accountability measures for 
postsecondary education in Florida. PEPC is currently working toward an approach acceptable to all 
interested parties. Part of the problem is understanding what is being sought; another is finding common 
language to talk about it. PEPC is currently circulating a draft concept paper designed to clarity terms, 
set a framework, and develop a statewide approach to accountability. Key terms are defined as follows: 

For purposes of this paper, accountability will be viewed as externally focused — providing 
information that responds to requests from such sources as state government and regulatory or 
licensing agencies. Assessment is internally focused — producing and using data and other types 
of information to evaluate processes and procedures in higher education. The results of 
assessment activities in a college or university can be used to respond to accountability 
demands. Because outcomes assessment has been perceived as a powerful force for change, 
specifically as a vehicle for improving the quality of education, legislators and educators have 
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focused on outcomes assessment as the way to achieve accountability as well as to enhance 
planning and quality improvement [italics in original]. 12 

The general conceptual approach posited by PEPC is stated below: 

Accountability is usually perceived as a top-down activity, where an agent external to the 
postsecondary institutions requires that those institutions be answerable for their activities. 
However, accountability should occur on two levels: the state level and the institutional level. 
Thus, both the State and the individual institutions are held accountable to the citizens of the 
State. The purposes of accountability for the State and the institution are related but are not 
exactly the same. At the state level, the primary purpose is oversight to guide the educational 
system toward improvement and achieve public reporting. At the institutional level, the primary 
purpose of accountability is to improve the effectiveness and efficiency of the institution. In the 
final analysis, accountability should direct institutions and the state toward improvement. 13 

To encourage consistency and common understanding in accountability, PEPC suggests an explicit 
state-level accountability policy designed around "a few broad, long-term goals and a few indicators that 
focus on what is most important and that show how we are doing as a State on those few goals." These 
goals call for policy (1) to be based on the mission statement of the state's postsecondary education 
system, (2) to be directed to a few specific state goals, (3) to provide key measures for each goal, (4) to 
include information that shows system performance at a single point In time as well as changes in 
performance over time, (5) to allow for comparison with peer systems, (6) to evaluate the quality of the 
process as well as the inputs and the products, (7) to include assessment of the achievement of general 
education objectives by undergraduate students, and (8) to be reported to the public in understandable 
format. 

The paper includes a discussion of accountability at the institutional level, recognizing the variety of 
accountability measures that have been used for years and supporting the need for institutional measures 
to be more numerous and variable in recognition of local needs and differing missions and goals. The 
general philosophy underlying state-level accountability efforts is summarized in the conclusion: 

Accountability should be an integral and continuous part of postsecondary education and 
another tool for educators and policy makers to use for informed decision making. 
Accountability should not drive education, but it should support education by drawing on 
information to respond to legitimate questions about what colleges and universities do and how 
well they do them. 
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As Florida struggles to develop a comprehensive state-level indicator approach, it seems likely thai 
whatever measures are finally adopted will include some form of comparative data among institutions. 
In addition to the PEPC concept paper, the State University System accountability plan explicitly 
recommends such an approach for its institutions: 

Florida's universities represent institutions in differing stages of growth, development, and 
mission so that simple comparisons within the state yield inaccurate evaluative results. Instead, 
each university should establish appropriate external institutional counterparts (e.g., a group of 
top ten peer institutions for each university) for whom comparisons can serve as a yardstick of 
performance. 14 

It is less clear what, if any, other kinds of contextual data will be incorporated. Given the emphasis on 
planning and goal setting among Florida educators, it also can be expected that accountability 
measures will be related to institutional and sector goals, as expressed in master plan documents 
currently being updated by the two systems and PEPC. With the base provided by the 1991 accountability 
statutes for each system, it also seems likely that both quantitative data and qualitative measures 
(espxially surveys of various constituent groups) will be used. 



CONNECTIONS TO OTHER PROCESSES 



Linkages to Budget 



The possible use of budgetary incentives or penalties is being discussed in relation to state-level 
accountability processes. Recent statements in the new iLister plan suggest several strategies that tie 
budgetary incentives to accountability. For example, in the section concerning instruction, the 
commission suggests ensuring that tenure, salary, and promotion criteria appropriately reflect 
institutional mission. In the management section, the plan proposes incentive funding to attain specified 
objectives in each sector or across sectors (e.g., increases in minority graduation rates, graduates in 
economic development occupational clusters, and salary increases tied to attainment of accountability 
goals). The commission also acknowledges the Division of Vocational, Adult and Community Education 
for its new performance-based funding model designed to provide incentive funding based on 
improvements in enrollments, completions, and placements. 
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Linkages to Planning 

There is an overt effort to fit accountability policy discussions within the context of overall 
planning. PEPCs 1992 report on outcomes assessment makes this link explicit: 

... the Commission does not recommend moving to such an assessment process at this time in 
light of the greater need to use the master plan process during the next twelve months to address 
fundamental issues of management, structure, productivity, and financing of Florida's 
postsecondary education system. A result of the Master Plan process should be a definition of 
the role of assessment and how it should be implemented in the State's postsecondary education 
system. 

This outcome was not realized during the master plan process. It is currently under discussion as part of 
PEPC's legislatively mandated study on accountability. 



CONCLUSION 

Florida was an early leader in efforts to assess student performance through statewide testing. It has 
been less on the cutting edge of efforts to assess institutional performance in a comprehensive state-level 
approach. In response to legislative initiatives, however, Florida has grappled with state-level 
accountability. PEPC's current project to design a state-level accountability process will not only provide 
information for policymakers interested in pursuing improvement but also data that will allow the public 
to judge how well their colleges and universities are performing against a set of state-level goals. The 
"Florida approach," however it turns out, will be of wide interest to other states searching for the right 
balance between external agency need for accountability and internal institutional need for improving 
performance. 



NOTES 

1. Unlike other states, Florida's State Board of Education is composed of seven members serving ex officio by 
virtue of the elective office he or she holds — the governor, secretary of state, state treasurer, attorney 
general, commissioner of agriculture, state comptroller, and commissioner of education. Constitutionally based, 
the board serves as the single governing board for all public education in the state. Source: Aims C. 
McGuinness and Christine Paulson, State Postsecondary Education Structures Handbook 1991 (Denver: ECS, 
1991). 
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. 2. Structures Handbook, 199 L 

3. Structures Handbook, 1991. 

4. Augenblick, Van de Water and Associates, "An Examination of the Overall Structure for the Delivery of Public 
Postsecondary Education in Florida," Final Report to the Structure Committee of the Florida Postsecondary 
Education Planning Commission (Denver: AVA, January 17, 1990), p. 6. 

5. Postsecondary Education Planning Commission, College-Level Academic Skills Test Review, 1988, Report 4. 
(Tallahassee, FL: Postsecondary Education Planning Commission, February 25, 1988), p. ii. 

6. College-Level Academic Skills Test Review, 1988. 

7. Post- econdary Education Planning Commission, Outcomes Assessment in Postsecondary Education, 1992, 
Report 5. (March 1992), p. i. 

8. Outcomes Assessment, p. A-l. 

9. Office of the Auditor General, Assessment of the State University System of Florida Accountability Plan 
(Tallahassee, FL: Office of the Auditor General, December 31, 1991). 

10. Horida Community College System 1992 Accountability Report, Appendix A. 

11. Postsecondary Education Planning Commission, Outcomes Assessment in Postsecondary Education: Report 
and Recommendations of the Florida Postsecondary Education Plaming Comtnission, 1992, Report 5. 

12. Postsecondary Education Planning Commission. "Draft Concept Paper: An Accountability Process for 
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ILLINOIS 



Richard C Richardson, Jr. 



OVERVIEW 

Planning and development of state-level indicators are linked closely in Illinois. Because these 
efforts are inextricably tied, it is difficult to document their evolution over time. Over a number 
of years the Illinois Board of Higher Education has established working groups of board staff, 
trustees, and institutional representatives to study such priorities as undergraduate education, work force 
needs, productivity, diversity, faculty roles and responsibilities, and affordability. In most cases, 
assessment measures are an integral part of these efforts. For example, recommendations of a 
productivity committee led to examination of institutional programs, which, in turn, resulted in 
recommendations to eliminate, consolidate, or reduce 190 programs. All assessment activities profiled in 
this case study were initiated by the coordinating board, which produces reports summarizing system 
and institutional effectiveness in identified areas. In many cases, national data sources are used to assess 
the health of Illinois higher education. 

Beginning in 1991, the board undertook the Priorities, Quality and Productivity (PQP) initiative, under 
which institutions are required to report annually on progress made in carrying out productivity 
improvements. The board also produces an annual report summarizing statewide progress on productivity 
goals. Even though Illinois is still in the beginning stages of state-level reporting, linking quality 
measurement with system planning has been successful thus far. To date, the coordinating board has 
remained ahead of the legislature and governor in initiating accountability efforts. 



CONTEXT 



System Design 

The Illinois system of higher education has 185 degree-granting institutions, 62 of which are publicly 
controlled. 1 Twelve public universities are controlled by 4 governing boards, each of which oversees at 
least 2 institutions. Forty-nine community college campuses are governed by 40 local boards of trustees 
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who work with the Illinois Community College Board in carrying out their missions. Illinois' public 
higher education institutions are coordinated by the 17-member Illinois Board of Higher Education 
(IBHE), which has statutory authority to develop master plans; approve new programs of instruction, 
research, or public service; review existing programs; recommend budgetary needs for operations and 
for capital improvements to the governor and general assembly; administer state and federal grant 
programs; and approve or disapprove operating and degree-granting authority for nonpublic 
colleges and universities. 2 

Student Demographics 

Illinois public and private institutions serve about 750,000 students each year and award 100,000 
certificates and degrees in a full range of academic specialties. Overall, the student body is racially and 
ethnically diverse, with African- American, Hispanic, and Asian-American enrollments increasing 
substantially over the past decade. This diversity is not common across institutions, however. 
African- American and Hispanic enrollments tend to be clustered at the community colleges and at a few 
urban four-year institutions including Chicago State University, University of Illinois at Chicago, and 
Northeastern University. 



SYSTEM EVOLUTION: INITIATIVES AND PRIORITIES 

During the 1960s and 1970s, IBHE developed master plans focused on improving access and health 
education. The 1976 Master Plan for Postsecondary Education in Illinois recognized the need for a new 
approach to planning. Future attention to goals and priorities was to be handled through a continuous 
process rather than by issuing successive versions of a master plan. After 1976, IBHE established ad hoc 
committees in defined priority areas and relied on them to develop recommendations for IBHE adoption. 
Among the topics considered during the following decade were teacher education, education in the 
health professions, off-campus programs, operating and degree-granting authority for nonpublic 
institutions, community college financing and public university tuition, student financial aid, higher 
education^ role in economic development, remediation, the teaching of writing, and baccalaureate 
admission requirements. 3 Since the mid-1980s, in an era of declining state fiscal support, IBHE has 
focused on five major goals: maintaining diversity, extending educational opportunities, assuring 
academic excellence, promoting economic development, and improving cost effectiveness. 
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Before turning to the central issue of undergraduate education, this case study examines the 
implications of board actions related to opportunities for underrepresented groups, work force 
preparation, and productivity improvements. 

Opportunities for Underrepresented Groups 

In 1985, IBHE established goals to improve minority student high school preparation and graduation 
rates and baccalaureate retention. There was a particular interest in increasing minority student 
enrollments in engineering, math, science, and in graduate and professional programs. In 1989, these 
goals were amplified when the board adopted guidelines for planning and reporting by public 
institutions. Each institution was required to submit an annual report detailing goals for improving 
participation and success rates for minority, female, and disabled populations; identifying strategies and 
programs for attaining goals; and reporting results. 

As of today, all public institutions within the system have formulated such plans. They typically include 
high school articulation and summer bridge programs, remedial and tutorial services, financial support, 
and the creation of supportive campus environments. Comparable strategies address the needs of women 
and the disabled. Most institutions have not created specific enrollment or employment goals for 
underrepresented groups. 4 Recent efforts to improve minority baccalaureate attainment have focused on 
improving transfer opportunities. 5 

Work Force Preparation 

The IBHE initiative on work force preparation was designed to prepare students at all levels of 
education for productive lives and careers. The initiative emphasized cooperation with business as well, 
as with other education and training providers. Strategies for achieving this goal included improving 
basic skills through cooperation with the K-12 sector and through adult basic education provided by 
community colleges, establishing centers for learning excellence in adult basic education at a number of 
community colleges, establishing regional or statewide centers for emerging technology, financing 
consortia of higher education institutions to assist small and medium size industries in employing 
advanced manufacturing technologies, developing technology preparation programs by promoting 
cooperation between community colleges and secondary schools, and establishing information 
clearinghouses to disseminate information on job training and employment services. 

Reporting procedures to monitor work force preparation include (1) a survey of public university 
graduates that collects information on the relationship between undergraduate education and 
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employment, (2) an employer feedback system to provide information on graduate success in the 
workplace (under development), and (3) a process for guaranteeing the skills of graduates of both the 
occupational and baccalaureate training programs (this is being developed). 6 

Productivity Improvements 

In response to gubernatorial concerns and mixed public perceptions about higher education 
accountability and productivity, IBHE established the Committee on Scope, Structure and Productivity 
in 1989. Defining productivity as including quality, cost effectiveness, and accountability, the committee 
identified four guiding assumptions for its work: (1) anything currently being done could be done better; 
(2) improvements should be accompanied by cost savings or, at the worst, no cost increases; (3) new 
programs and services should be financed by greater efficiency in current operations or by the 
termination of less important activities; and (4) results should be reported regularly. 

In response to the committee's work, IBHE adopted comprehensive strategies. Campus leaders were 
asked to involve entire campus communities in broad-based efforts to align institutional goals and 
priorities with those articulated by IBHE and to design strategies to improve campus quality and cost 
effectiveness. Three reports were identified as essential to keeping track of results. Institutions were 
asked to prepare a "consumer report" to inform current and prospective students and their parents about 
student success and satisfaction; they were also to create a "productivity report" to inform the campus 
community and state officials about quality and cost effectiveness efforts. IBHE prepared the third report 
to inform the public at large about the goals and conditions of Illinois higher education. 7 

IBHE experimented briefly with these reporting requirements, but all three were replaced under the 
newly adopted Priorities, Quality and Productivity (PQP) initiative. Currently, each institution produces 
a productivity report detailing campus progress and an annual board report summarizing statewide 
progress on PQP goals. Under PQP, IBHE constituted itself as a committee of the whole, identifying five 
initial areas to be studied for productivity improvements. The higher education sectors were asked to 
carry out an in-depth review of each area and share their results with IBHE. The budget process was 
identified as the key strategy for addressing priorities and making productivity improvements. Providing 
institutions with incentives for making productivity improvements and flexibility in implementation was 
a key part of the IBHE approach to what was a very sensitive process. 8 

The particular areas to be studied were instruction, research and public service, overall academic 
functions, administrative functions, and state policies affecting higher education. Specific guidelines for 
reviewing instructional units included student demand, occupational demand, centrality to mission, 
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breadth of offerings, quality of the instructional unit, success of graduates, and program costs. 
Institutions were expected to use quantitative and qualitative information available to them in examining 
the priorities, quality, and productivity of their instructional programs, as well as other aspects of their 
operation. IBHE provided information about student demand, degree production, costs, and a measure of 
centrality for groups of related programs, with suggested ratings of high, moderate, and low to 
encourage institutions to consider their offerings in relation to those of other colleges and universities. 9 

By August 1992, IBHE had determined that "productivity improvements ... are achieved by 
eliminating low quality and low priority programs and activities ... to redirect resources to high 
priorities." To move the process along, the board established 25 guidelines for making improvements in 
the five key areas. To guard against the possibility that its actions might be interpreted as insensitive to 
institutional diversity or too directive for a coordinating board, the accompanying discussion recognized 
that campuses needed flexibility to achieve improvements consistent with individual institutional 
missions and priorities. Each system and institution was encouraged to weigh collective findings on all 
guidelines against the judgments of faculty members, administrators, and board members to arrive at 
decisions about where improvements could be made and when. 10 

At the October 1992 board meeting, based on an analysis of the productivity of instructional units, 
IBHE staff recommended the elimination, consolidation, or reduction of 190 programs at public 
universities, including 7 percent of all undergraduate programs. In addition to reductions in instructional 
programs, staff recommendations called for phasing out state support for intercollegiate athletics, 
redirecting 6 to 9 percent of the expenditures for research and public service to undergraduate education, 
and paying particular attention to faculty workload reassignments to undergraduate education. 

The object of the recommendations was to free 6 to 8 percent of current resources, which would be 
retained by each university to reinvest in such state priorities as strengthening undergraduate teaching 
and learning, enhancing minority student achievement, keeping higher education affordable, improving 
faculty and staff salaries, and addressing deficiencies in library materials, instructional support, and 
facilities repair and maintenance. 11 

During the October meeting, public universities and the community college system presented reports 
outlining productivity improvements during the previous year. The IBHE plan for the following year 
called upon institutions (1) to implement fully the productivity improvements described in their 1992 
reports, (2) to immediately engage campus communities in full consideration of IBHE staff 
recommendations for program reductions, (3) to implement the recommendations or comparable 
alternatives, and (4) to work cooperatively with board staff on priorities for 1992-93. These priorities 
included off-campus programs, state policies and reporting procedures, graduation rates and 
time-to-degree, transfer and articulation, academic calendars, faculty workload, and student financial aid. 1 ' 
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By September 1993, institutions had submitted productivity reports summarizing their responses to the 
1992-93 plan. Forty-four percent of the identified 190 programs were eliminated or reduced, 26 
percent remained under continuing review, and 30 percent were retained. Most institutions developed 
plans to phase out state support for intercollegiate athletics, but some replaced state funds with increased 
student fees. The IBHE progress report on the PQP initiative included narrative summaries analyzing the 
progress of institutions in identifying recommended next steps for the following year. During 1993, 
IBHE added the issues of affordability and faculty roles and responsibilities to institutional agendas. 13 



UNDERGRADUATE EDUCATION 



In September 1986, IBHE adopted the recommendations of the Committee on the Study of 
Undergraduate Education. The committee called for commitment to three areas: student preparation and 
remediation, student achievement and scholarship and general education, and faculty teaching 
excellence. The committee report proposed community colleges to be the principal source of remedial 
education, reaffirmed the board policy of no credit toward degrees for remedial courses, and proposed 
the creation of an information system to foster cooperation among the public schools and higher 
education institutions. 

To improve student achievement and general education, the committee also proposed increasing 
faculty/student interaction in scholarly and community service activities, enhancing student commitment 
to learning and undergraduate achievement, emphasizing the centrality of general education, and 
developing the means to assess student progress, The committee further urged IBHE to give priority to 
the attainment of these goals in the program approval and review process and in budget development. 

Finally, the committee emphasized a series of recommendations that focused on the faculty: time 
devoted to undergraduate teaching should be recognized through salary promotion and tenure, the best 
and most experienced faculty should teach undergraduates, institutions should ensure that faculty are 
well prepared to teach and be proficient in English, institutional planning and resource allocation 
policies should emphasize undergraduate education and faculty development, and the program review 
process should focus on instructional improvement. 14 

The Committee on Undergraduate Education was reconvened in 1989. After meeting and 
deliberating, it presented to the board 32 recommendations that were adopted in 1990. The new 
policies reaffirmed much of the committee's earlier work. To the three areas previously identified for 
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emphasis* the committee added transfer and articulation. Remediation was dropped, with the first area of 
focus redefined to include student preparation, access, and retention. Six statewide goais for 
undergraduate education were identified: (1) strengthening the academic preparation of high school 
students, (2) expanding access and improving students' chances for success, (3) promoting excellence in 
undergraduate teaching and increasing student/faculty interaction, (4) enhancing student involvement in 
and commitment to learning, (5) emphasizing the centrality of general education, and (6) establishing 
partnerships between two- and four-year institutions. 

IBHE policies on undergraduate education call for annual status reports on the progress made by public 
universities and community colleges in implementing improvements. In these reports, institutions are 
expected to examine trends in undergraduate retention and graduation rates; class size, faculty and 
course evaluations; faculty/student interactions in and out of class; student use of and satisfaction with 
academic and support services; resource allocation to undergraduate instruction; employment rates, 
enrollment in further education, and success in certification or licensing examinations; and special 
analytical or longitudinal research studies. During 1992, the IBHE staff produced reports on progress in 
implementing its undergraduate education policies. The first report on teaching and learning reported 
five statewide trends. 15 The second report, Undergraduate Education: Access and Preparation, revealed 
much about the political process through which higher education policies are negotiated in Illinois. 

State-Level Indicators 

Many of the indicators used by Illinois in assessing productivity and quality followed logically from the 
board's design for improving the quality of undergraduate education. These are summarized in table 1. 
IBHE has designed or is designing four statewide information systems to collect the data used in 
producing indicators of system and institutional performance. Data provided by these systems are 
supplemented by institutional data collection and analysis such as the follow-up surveys individual 
community colleges undertake of the graduates of their career-oriented programs. 
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TABLE 1 

IMPROVING THE QUALITY OF UNDERGRADUATE EDUCATION 



Preparation/Entry Undergraduate/Exit 



Establish Standards 


Admission Requirements: 


Graduation Requirements: 


and Expectations 


• Minimum high school course requirements 


- General Education 




• Entrance Exaias (ACT, SAT) 


- Baccalaureatc-level skills 




• Class rank and grade point average 


- Program map*' 






- Minimum credits and grade point 








Publicize Standards 


• Statement on "Preparing feu College'' sent to 


• College and university catalogs 


and Expectations 


high school students via superintendents. 


• Academic program brochures 




principals, counselors, v\6 board presidents 


♦ New student orientation programs 




• Learning Outcomes for Colkge-Bound Students 


• Academic advising 




• College and university catalogs, admission 






brochures, and application forms 






• College and university admissions counselors 






working with high school counselors 






• College Nights/Fairs 




Assist in Meeting 


• School-College Partnerships to improve education 


* Academic advising 


Standards 


• Early outreach programs for at-risk and advanced 


* Entry-level skills assessment for 




students 


course placement 




* Summer bridge or transition programs 


* General education and major courses 




* Teacher preparation and in-service training 


* Honors programs 




• High School Feedback System reports 


* Assessment of whether 






baccalaureate-level skills and of 






objectives in general education and 






the major have been achieved 






• Provide guidance through: 






- Tutoring programs 






- Mentoring Programs 






- Supplemental sections in gateway 






courses 






- Workshops on study skills, note 






and test taking and time 






management 






- Career exploration and planning 






programs 



Source: Dlinois Board of Higher Education. Undergraduate Education. March 1992, Item a. 
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Recent status reports produced by IBHE provide an overview of the evolution of indicators within the 
system: 

1. Annual Report on Underrep resented Groups Mandated by Public Act 84-726 

This report includes trends in representation and enrollments for minorities, females, and 
disabled students; trends in employment for minorities and females; trends in degree 
completion and retention for minorities, females, and disabled students; efforts to improve 
representation; and statewide strategies, programs, and funding. 

2. Status Report on the Implementation of Board Policies on Underserved Areas 

This report distinguishes authority to operate at off-campus locations from authority to offer 
off-campus degrees and provides an inventory of authorized off-campus activities. The board 
policies are, among other things, designed to improve access, strengthen quality, improve 
coordination, and provide programs in a cost-effective manner. 

3. Status Report on Implementation of Board Policies on Preparation of the Work Force 

This report integrates and expands existing board policies in such areas as basic skills 
preparation, information clearinghouses, annual reporting on underrepresented groups, alumni 
follow-up, and employer feedback. 

Prior to the 1990 Report of the Committee on Scope, Structure and Productivity in Illinois Higher 
Education, 16 IBHE's practice was to report indicators related to its goals (diversity, access, academic 
excellence, economic development, and cost effectiveness) in an annual staff paper. 17 The concepts and 
proposals eventually made their way into an annual revision of the board's master plan. 18 Presumably, 
this planning and review process subsequently influenced institutional budget requests as well as the 
budget review and recommendation process by system boards and IBHE. 

The indicators reported in the 1990 Planning, Program, and Budget Issues document provide a frame of 
reference for the following review of the current reporting structure. Indicators summarized by topical 
area included the following: 

Maintaining Diversity — institutions, students, programmatic responsiveness, adequacy of support 

Extending Educational Opportunities — affordability, opening opportunities for 
underrepresented groups, opening opportunities in underserved areas 
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Assuring Academic Excellence — academic preparation, undergraduate student achievement, faculty* 
and excellence in teaching, program review, and school-college partnerships 

Promoting Economic Development — work force preparation and occupational trends, research, and 
public service 

Improving Cost Effectiveness — expenditures by function, personnel expenditures, and facilities 



Current Reporting Process 

The process described above has been supplanted by IBHE's work on priorities, quality, and 
productivity. The current effort calls for the production of two reports: a productivity report from each 
institution and a board report summarizing institutional progress in implementing PQP principles. 

The productivity reports present information from each institution to increase the outside visibility of the 
planning process. The first section provides information about institutional goals and their 
relationship to state goals, plans to improve quality and cost effectiveness, and descriptions of how 
resources will be allocated to further institutional goals and priorities. The second section provides a 
summary of the extent to which goals for the preceding year have been achieved. The reports are 
expected to include descriptions of improvements in quality and cost effectiveness, as well as 
descriptions of actual resource allocations and reallocations compared to those planned in the previous 
report. 

The board report informs the public about the performance of higher education statewide in relation to 
state goals and policy objectives. Indicators provided in the first of these reports include capacity in 
relation to student demand; capacity in relation to occupational demand; centrality in relation to 
institutional mission; breadth of instructional unit (majors/specializations in relation to enrollment); 
quality of the instructional unit (as determined through the program review process, which examines 
such variables as clarity of program objectives, faculty qualifications and productivity, curriculum, 
academic support services, student achievement and success, student enrollment levels, unit costs, and 
occupational demand); success of graduates (employment trends, further education, satisfaction, and 
pass-rates on licensure exams); and program costs that deviate markedly from the norms. The 1993 
report includes an assessment of progress in responding to the recommendations in the 1992 report, as 
well as a narrative section tracing the actions and work remaining for each institution. 
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CONCLUSION 



The Illinois experience is important to states implementing state-level indicators. First, Illinois has 
consistently sought indicators of progress in achieving the goals and priorities advanced by its governor, 
legislature, and by IBHE. The legislature provided incentives for IBHE to examine remediation, student 
preparation, and the status of underrepresented students. More recently, the governor has expressed 
interest in productivity and accountability. On its own initiative the board has examined issues related to 
work force preparation, underserved areas, affordability, and faculty roles and responsibilities. Each 
initiative has contributed to the current reporting system and the indicators that are emphasized. 

Second, Illinois* approach to establishing state indicators is inextricably linked to the planning, review, 
and budge^ng processes. IBHE was established to plan, review, and approve programs and 
recommend budgets. To the extent that these powers can be used in mutually supportive ways to pursue 
a limited number of priorities about which there is some public consensus, the board's authority is 
enhanced. Increasingly, IBHE has sought synergy among its highest priorities by uniting them within its 
initiatives on undergraduate education and productivity. 

Reporting is the key to using indicators effectively. Ideally, reports are designed specifically for an 
intended constituency, appear with regularity, and are used to mobilize the general public as well as state 
policymakers in improving institutional accountability. Technical reports prepared by a state agency for 
the internal use of board members are no substitute for the institutional productivity reports currently 
being produced in Illinois. 

Coordinating boards, even strong and experienced ones, walk a difficult path in defining and applying 
performance indicators. When boards are perceived as overly zealous, they invite legislative 
intervention. When they are overly cautious, institutions remain intransigent and non-responsive. Any 
attempt to establish indicators for specific institutional responses can benefit from the cautious Illinois 
pattern of building consensus, planning for change, allocating resources primarily in the form of 
incentives, requiring reports, and implementing stronger central intervention only after unsatisfactory 
institutional responses have created impatience among policymakers and the general public. 
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KENTUCKY 



Sandra S. Ruppert 



OVERVIEW 




entucky's 1990 assessment mandate, the Kentucky Accountability Enhancement Program 
(KAEP), was preceded by several years of higher education academic program review and 



m 3k» realignment. While the stated purpose of KAEP, which coincided with elementary and 
secondary education reform legislation, was to "respond to the public interest in the outcomes of higher 
education," the law closely paralleled early efforts in other case-study states that allowed institutions 
to develop their own assessment and reporting plans. 

In 1992, the general assembly passed legislation requiring public higher education institutions to 
report annually on 14 common performance standards. Patterned on legislation in such states as Florida 
and South Carolina, the measures were designed to monitor performance in the areas of instruction, 
research, and public service. In 1993, as part of an initiative to restructure state government, the 
governor appointed the Higher Education Review Commission (HERC) to conduct a thorough 
evaluation of Kentucky's higher education system. The HERC final report, presented to the governor in 
December 1993, contained a series of recommendations that included implementing performance-based 
evaluation and funding. If performance-based funding passes the general assembly during the 1994 
legislative session, addition-to-base funding will be awarded according to performance criteria. 



CHRONOLOGY AND CONTEXT 
System Design and Evolution 

The Kentucky Council on Higher Education (CHE), established in 1934, is considered to be the 
country's oldest state coordinating agency for higher education. Following substantial amendments over 
the years, council duties and responsibilities currently include comprehensive planning, institutional 
budget review, and consolidated recommendations for state funding of higher education. Further, CHE is 
authorized to define and approve the offering of all degree programs, determine tuition levels, and set 
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minimum admission requirements at the public institutions. The council also has responsibility for the 
review and approval of all capital construction and renovation projects exceeding $400,000. 

Statutes defining council membership were revised in 1980 to reflect the current composition, which 
includes 17 lay members appointed by the governor. The commissioner of the department of education 
also serves as an ex-officio member. Kentucky's statutes provide that among the lay members, there 
must be at least one undergraduate degree recipient from each of the state universities and one member 
from each of the seven supreme court districts, thus ensuring regional representation. Although the state 
has a Secretary for Education, Arts and Humanities, the council is an independent board, responsive to 
the legislature but reporting directly to the governor. 

Under Kentucky's governance system, each of the eight public universities has an institutional 
governing board. The University of Kentucky's Board of Trustees governs the institution in addition to 
14 public community colleges. Seven other boards each govern a single institution: University of 
Louisville, Eastern Kentucky University, Kentucky State University, Morehead State University, Murray 
State University, Northern Kentucky University, and Western Kentucky University. The Council of 
Independent Kentucky Colleges and Universities serves as the state-level organization for 20 of 
Kentucky's accredited nonpublic colleges and universities. 

Selected Facts and Trends Related to Higher Education in Kentucky 

A number of forces are affecting Kentucky higher education at this time: 

• High school graduation rates and college-going rates have increased in the last decade. However, at 
the same time, the actual number of high school graduates has declined since 1978 and is expected 
to decline through 2000. 

• Beginning in fall 1986, headcount enrollment at Kentucky's public and independent institutions has 
grown steadily, resulting in a 38 percent increase by 1992. The most significant increase (102 
percent) was at the University of Kentucky community colleges. Public university enrollment 
increased by 26.1 percent and enrollment at independent institutions increased 13 percent between 
1985 and 1992. 

• State supported universities saw a 4 percent decrease in first-time freshmen enrollment between 
1982 and 199 U while total enrollment increased by 108 percent at community colleges. During the 
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same time period, African-American enrollment increased by 26 percent at state supported two- and 
four-year institutions. 



CHRONOLOGY OF RELATED PRIORITIES AND INITIATIVES 

1981: CHE's blue-ribbon citizen group, the Prichard Committee, releases a seminal study, In Pursuit of 
Excellence. The committee issues 100 recommendations, the vast majority of which are under CHE's 
jurisdiction. In the following five years, complete review of all degree programs is conducted, resulting 
in a decrease of about 200 programs; a precollege curriculum is established; the tuition policy is revised 
and geared to the ability of students to pay; a formula-funding system is developed; and implementation 
is begun on a systemwide plan for desegregation of higher education. 

1985: The council adopts its first five-year plan, the Strategic Plan for Higher Education in 
Kentucky, which contains 8 goals and 50 objectives. 

1989: The council revises academic program review guidelines to strengthen connections with the 
strategic plan. The new guidelines emphasize qualitative assessment and program improvement in all 
decisions relating to academic program improvement. 

1990: The Kentucky Education Reform Act (KERA) is passed, which commits the state to 
comprehensive and fundamental changes in its system of elementary and secondary education. A 
concurrent resolution of the general assembly gives higher education roles and duties in this process. 

1990: Based on recommendations from the governor and CHE, the general assembly creates and funds 
the Kentucky Accountability Enhancement Program (KAEP), designed to achieve systemwide 
improvement in higher education. 

May 1991: The second Strategic Plan for Higher Education in Kentucky 1991-1996: Vision Focus 
Action is adopted. Themes include planning, cooperation, articulation, teaching, education reform, 
technology, funding, and accountability. 

February 1992: The general assembly passes Senate Bill 109 (SB 109), outlining 14 performance 
standards for public higher education institutions and requiring annual reporting of progress on these 
standards. 
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September 1992: CHE produces the Kentucky Plan for Implementing the Higher Education 
Accountability Process, otherwise known as the Accountability Process Plan., for reporting on each of 
the 14 performance standards outlined in SB 109. 

September 1993: Governor Brereton Jones appoints a higher education review commission to 
conduct a thorough evaluation of higher education and report its findings by December 1993. 

November 1993: The council authorizes a comprehensive review and revision of the higher 
education funding formula during 1994-95 and recommends implementation of performance-based 
funding in the 1996-97 biennial budget cycle. 

December 1993: The governor's Higher Education Review Commission (HERC) adopts its final report 
and recommends performance funding for 1995-96. The governor includes this approach in his 
executive budget recommendation in January 1994. 



ASSESSMENT OF INSTITUTIONAL EFFECTIVENESS 



Over the past decade, Kentucky's approach to assessing institutional effectiveness has been linked to 
systemwide and institutional planning. The state's history of strategic planning began with the 1981 
report of the Prichard Committee on Higher Education in Kentucky's Future. Their report. In Pursuit of 
Excellence, resulted in Kentucky's first five-year strategic plan, adopted in 1985. The council completed 
a second cycle of strategic planning and published its report in 1991. While access was the primary 
focus in the earlier plan, quality issues, particularly related to undergraduate education, surfaced as the 
focus of the second. 

Extensive academic program reviews were undertaken during the 1980s. Strategic plans increased 
expectations for such reviews, emphasizing instructional improvement and greater reliance on 
qualitative assessment of program outcomes. New guidelines published by the council in 1990 also 
strengthened the program-review process by merging program review and planning functions. In this 
merged process, institutional program review was geared to providing greater support to institutional 
planning and priorities. 

Legislative actions in 1990 were to have a significant impact on Kentucky's education system. That 
year, the Kentucky legislature passed the Kentucky Education Reform Act (KERA), which committed the 
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state to making comprehensive and fundamental changes in its elementary and secondary education 
system. It was later to shape higher education developments as well. 

For higher education, lawmakers created and funded the Kentucky Accountability Enhancement Program 
(KAEP). The stated purpose of KAEP was to "respond to the public interest in the outcomes of higher 
education," placing the focus squarely on accountability. However, the law closely paralleled earlier 
institutional assessment efforts in other states by specifying that institutions should develop their own 
assessment and reporting plans. The program sought to establish "a coordinated system-wide effort to 
demonstrate and improve the effectiveness of higher education in meeting Kentucky's needs." 

Under KAEP, each public institution was to conduct an ongoing strategic planning process specific to its 
mission and responsive to systemwide goals. Institutional accountability would be achieved through 
attention to the mission and goals outlined in strategic plans. System-level accountability was directly 
tied to the goals generated through the council's systemwide planning. 

KAEP was seen as a way to coordinate a number of existing efforts aimed at improving higher 
education. Through this process, "institutional progress and target areas in need of improvement" could 
be documented. As part of a larger strategic planning process, KAEP was intended to provide new types 
of information to help determine the areas in which higher education was improving and the areas where 
change was needed. 

Process Required 

The council adopted a collaborative approach in designing the process for implementing KAEP. The 
Kentucky Accountability Network was formed to solicit institutional input during the development of 
KAEP guidelines as well as to provide an avenue for ongoing communication between the council and 
Kentucky's colleges and universities. The network also shared information related to the development 
and implementation of campus-based assessment programs. An advisory board for the network, 
comprised of council staff and institutional representatives, was charged with planning the direction of 
assessment and accountability in Kentucky. 

In addition to the Kentucky Accountability Network, during 1990-91 an office was established within 
CHE, charged to offer technical assistance for institutional assessment activities and to facilitate 
statewide conferences. Further assistance was supported by funding in the council's budget for 1990-92, 
earmarked for the development of model approaches to assessment. During that time, the council awarded 
two rounds of mini-grants for institutions to conduct pilot studies and implement student assessment 
practices. 
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Assessment Domains 

The approach taken to implementing KAEP was campus-based and respected distinct 
institutional missions. Each institution developed an assessment program to monitor efforts to meet 
goals and objectives identified through its own strategic-planning process. At the same time, each 
institution defined program quality for the purpose of academic program review activities, developed 
criteria consisting of resources and student/program outcome measures by which program quality would 
be assessed, and reported the findings and recommendations to the council. 

The council and institutions shared the responsibility for monitoring performance levels and 
determining the adequacy of existing academic programs, in the process identifying and resolving areas 
of concern. The program review guidelines specified only that institutions should monitor productivity 
levels, avoid unnecessary duplication, and address discipline-specific concerns. The latter included 
manpower shortages and surpluses, graduate survey findings, rate-of-pass on licensure examinations, 
job placement rates, rate of enrollment growth, and staffing and equipment availability. 



STATE-LEVEL INDICATORS 



In Governor Jones' January 1992 State of the Commonwealth Address, he spoke of the need to "adopt 
the same principles in higher education that made our primary and secondary education reform efforts 
the model for the nation." He suggested that higher education needed to have goals in instruction, 
research, and public service, as well as a system for assessing each institution's progress. 

That year, the Kentucky legislature passed Senate Bill (SB 109), which established specific 
reporting requirements for the state's public colleges and universities. The measures were 
designed to monitor performance in the areas of instruction, research, and public service while still 
recognizing individual institutional missions. To implement this bill, the legislature appropriated funds 
directly to colleges and universities. 

The specific language of SB 109 included the following directives: 

... an accountability process [is to] be implemented which provides for a systematic, ongoing 
evaluation of quality and effectiveness in Kentucky public institutions of higher education and 
to provide a method for evaluating each institution's progress toward meeting specific standards. 
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It is further the intent of the General Assembly that the accountability process monitor 
performance at the institutions in each of the major areas of instruction, research and public 
service while recognizing the individual missions of each of the institutions. The accountability 
process shall provide for the adoption of systemwide and individual performance goals with 
standards identified through a collaborative effort involving higher education institutions and the 
Council on Higher Education. 

In September 1993, Governor Jones appointed the Higher Education Review Commission (HERC) to 
conduct a thorough evaluation of Kentucky's higher education system as part of an initiative to 
restructure state government. The commission, headed by the CHE chair and composed of public 
university presidents, board chairs, key legislators, governor's staff, and the council's executive director, 
was charged to submit to the governor a plan for restructuring public higher education that focused on 
"specialization, quality, performance, cooperation and controlling expenditures." Governor Jones 
described the work of the commission as a "fact-facing, not a fact-finding" effort. 

In a speech to HERC, the governor offered higher education institutions the opportunity to be spared 
budget cuts beyond the more than $80 million they had experienced since October 1991. In return for 
avoiding further cuts, higher education was to address quality and efficiency issues through cooperation 
and within available resources. 

The governor outlined 14 topics for the commission's attention. These included mission-statement 
refinement; program duplication elimination; student performance measurement; campus management 
evaluation; refinement of efficiency goals, including the review of administrative and faculty workloads; 
and improving coordination statewide. Particularly relevant was the recommendation that HERC closely 
examine how quality and efficiency in higher education could be promoted through performance-based 
evaluation and funding". 

HERC presented its final report to Governor Jones on December 20, 1993, two weeks prior to the 
opening of the 1994 legislative session. In attempting to respond to the governor's challenge "to remake 
higher education in Kentucky," the HERC report contained 14 recommendations relating to specific 
areas of gubernatorial interest. Three recommendations addressed developing and implementing a revised 
funding model based on performance measures, many of which were already reported under SB 109. 

Process Required 

SB 109 established the prioritized measures for institutional accountability. The bill explicitly 
identified data to be collected and set out the process and timeline for gathering them. The Kentucky 
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Accountability Committee was then responsible for developing the plan and the process for 
implemention. The result was the Kentucky Plan for Implementing the Higher Education Accountability 
Process, which recognized parallel processes for institutional and system-level accountability. 

Collectively, the two processes represented systemwide accountability. The KAEP process was similar 
but more explicii ..bout goal setting and reporting relative to SB 109 criteria. Four phases of the process 
were identified: 

Strategic planning — Systemwide strategic planning was completed with publication of the current 
strategic plan. Instituu jns were expected to conduct strategic planning and set priorities according to 
their missions and the council's plan. 

Baseline reporting — During this phase, systemwide and institution-specific information were to be 
collected and analyzed according to the 14 indicators in SB 109. 

Goal setting — After the establishment of baseline reporting, performance goals were to be 
developed for both the system and institutions relative to SB 109 criteria. 

Annual reporting — Beginning December 1, 1993, CHE was to submit to the governor and the 
legislature an annual accountability report that included information on implementation of the 
mandated standards, achievement of goals, and initiatives to be undertaken. 

In the December 1993 proposal, HERC agreed to a revised funding model based on the following 
principles: the need for funding beyond the core level to be outcomes-driven (i.e. performance-based), 
the use of incentives to achieve desirable management goals, and the increased use of institutional 
mission differentiation rather than extensive use of "common funding for common activities." In terms 
of the set of measures to be used, HERC recommended that they address the following areas: 
persistence of students, student outcomes, quality of instructional programs, quality of research/service 
programs, and campus management. It was also agreed that measures in each of the areas should be 
based, when possible, on the data collected through SB 109 and the Higher Education Equal 
Educational Opportunities (EEO) Plan. HERC further reconfirmed that the measures should reflect 
mission differentiation among Kentucky's colleges and universities. Other goals and measures as needed 
would be developed cooperatively by the council and the institutions in future biennia. Moreover, HERC 
agreed to an increased connection between funding and quantifiable measures of success: 

. . . points should be awarded proportionately for progress tov/ard established goals in other than 
"evidence of or nonquantifiable measures. It was also recognized that reliance on "evidence of 
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measures, while necessary where baseline data are not available in the initial round of performance 
funding, would be reduced in future years. 

Assessment Domains 

Each institution was expected to respond to SB 109 by collecting indicators according to its mission 
statement, goals, and overall environment. These included the following: 

1. Total student credit hours produced, by institution and discipline 

2. Total number of degrees awarded, by institution and discipline 

3. Total number of contact hours of instruction produced by faculty, rank of faculty, institution, 
and course level 

4. A measure of faculty workload to include the hours spent in the following activities: 
instruction, course preparation, noninstructional student contact, research, and public service 

5. Pass-rates on professional licensure examinations, by institution 

6. Institutional quality as assessed by follow-up surveys of alumni, parents, clients, and employers 

7. Length of time and number of academic credits required to complete an academic degree, by 
institution and degree 

8. Enrollment, persistence, retention, and graduation rates by discipline and by race, gender, and 
disability 

9. Student course demand analysis 

10. Classroom utilization 

11. Research and public service activities, including activities supporting elementary and secondary 
education reform 

12. The number and percentage of accredited programs and the number and percentage of 
programs eligible for accreditation 

13. The percent and number of students enrolled in remedial courses and the number of students 
who exit remedial courses and then successfully complete entry-level curriculum courses 

14. The number of full-time transfer-students from a two-year, postsecondary institution and the 
number of these students who have successfully completed a four-year program 

HERC provided a list of performance-funding measures to be used by each institution along with the 
required minimum and maximum distribution of points in the five major areas for the 1995-96 
performance-based funds: 

Persistence of students (subtotal 10-30 points) 

1. Persistence of first-time, full-time degree-seeking freshmen (SB 109) 
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2. Persistence of African- American first-time, full-time freshmen (SB 109) 

3. Number of degree-seeking transfer students sent by community colleges to universities or 
received by universities from community colleges (SB 109) 

4. First-to-second year retention rate of first-time, full-time degree-seeking freshmen (SB 109) 

5. First-to-second year retention rate of African- American students (SB 109) 

Student outcomes (subtotal 10-30 points) 

1. Graduation rates (SB 109) 

2. Graduation rates of African- American students (SB 109) 

3. Success rates of students who complete remedial math and English courses, and successfully 
complete entry-level courses (SB 109) 

4. Satisfaction of graduating students (SB 109) 

5. Satisfaction of alumni (SB 109) 

6. Satisfaction of employers (SB 109) 

7. Periodic, comprehensive student assessment 

Quality of instructional programs (subtotal 10-30 points) 

1 . Evidence of elimination of programs targeted by HERC and evidence of periodic review of 
programs 

2. Percentage of faculty hours allocated to instruction (including course preparation and 
noninstructional student contact) (SB 109) 

3. Quality of incoming class by ACT score 

4. Success rates on licensure exams (SB 109) 

Quality of research/service programs (subtotal 10-30 points) 

1. External research support per full-time faculty (SB 109) 

2. Evidence of Kentucky Education Reform Act (KERA) activities (SB 109) 

3. Evidence of faculty involvement in service mission activities other than KERA — including 
economic development (SB 109) 

4. National ranking among U;S. research universities (NSF) 

Campus management (subtotal 10-30 points) 

1. Percentage of African- American employees in selected EEO categories (EEO plan) 

2. Percent of funds expended for institutional support (IPEDS) 

3. Adoption of strategic plans and evidence of program priority-setting 

4. Evidence of periodic update of a campus's facilities maintenance plan as envisioned by the 
Capital Planning Advisory Board 
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5. Evidence of periodic review of organizational structure 

6. Evidence cf the existence of a comprehensive staff-development plan 

7. Classroom and class lab utilization rates for degree credit activity (SB 109) 



REVIEW AND REPORTING OF RESULTS 
Comparative Use Data 

The guidelines in the accountability process plan for SB 109 required annual reports including a 
"concise, straight-forward systemwide summary report and brief, to-the-point, institution-specific 
reports." In the individual reports, institutions are to indicate individual progress toward their respective 
goals rather than to be compared with one another. Aggregate data are to be used to account for overall 
outcomes in the system-level report. 

The language of SB 109 recognizes distinct institutional mission differences but still requires that all 
institutions report on a common set of indicators. Thus the Kentucky Accountability Committee has 
adopted a reporting strategy that includes 22 institutionally specific reports (the 8 university and 14 
community colleges reports) and an overall systemwide summary. Under performance measures 
recommended by HERC and being considered by the general assembly for 1995-96, institutions will be 
eligible for performance funds if they report on a core set of outcomes. Preapproved standards will 
weight the performance of individual institutions in defined areas. 

Use of Quantitative and Qualitative Data 

Each SB 109 performance standard is classified as quantitative or qualitative. Quantitative indicators are 
total credit hours, total degrees, enrollments, persistence and graduation rates, transfer rates, 
instructional contact hours, faculty workload, time and credits-to-degree, course demand, and 
classroom utilization. Qualitative indicators include results from surveys of alumni, parents, clients, and 
employers; success following remedial courses; licensure-exam pass-rates of graduates; research and 
public service; and program accreditation. Performance-funding measures are primarily quantitative and 
overlap significantly with data collected through SB 109. 
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Use of Contextual Data 



As discussed above, strategic plans and institutional mission statements have been seen as providing the 
frameworks within which institutions can address system and institutional goals and then report their 
results. Institutional and system-level strategic plans are to be the vehicles for periodic reporting in order 
"to demonstrate more clearly to the general public and state policymakers that higher education is 
committed to reporting outcomes and able to account for the wise use of state resources." Results from 
program review, accreditation, and other campus-based evaluative processes are to be used as part of the 
evidence of institutional effectiveness. Initial emphasis has been placed on using existing information to 
determine how well the goals of strategic planning have been accomplished. In addition, the council is 
to prepare semi-annual system-level status reports that described improvement toward meeting 
systemwide goals. 



CONNECTION TO OTHER PROCESSES 
Linkages to Budget 

Although Kentucky higher education has lost ground in recent years in terms of the percentage of the 
total general fund it receives, special funding has been allocated specifically for accountability 
purposes. Under KAEP, the legislature provided additional funding to the council. In 1992, with passage 
of SB 109, the legislature for the first time appropriated some of the accountability funds directly to the 
universities. The 1992-94 budget also appropriated monies to the council for maintaining the 
accountability office, coordinating SB 109, and upgrading the council's computer system. 

As of this writing, the higher education general appropriations act, which included provisions for 
performance-based funding, had been approved by the General Assembly's House Appropriations and 
Revenue Committee and was headed to the entire general assembly for a vote. As detailed in the 
appropriations act, performance funds not distributed are to be returned to the general fund; additional or 
alternative performance measures will be considered in the revised funding model to be implemented in 
1996-98. 
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Linkages to Planning 

Kentucky's state policymakers have made conscious and deliberate efforts to link closely 
educational effectiveness and planning functions at both the state and institutional levels. This 
connection is basic to the state's concept of systemwide accountability. While the 1985 strategic plan 
emphasized access and attainment, the 1991 plan focused on quality as it pertained to both the "lives of 
Kentuckians and the public institutions of higher education." The focus of the 1991 strategic plan was 
on six goal clusters: quality, effectiveness, equal opportunities, attainment, elementary and secondary 
education, and economic development. The themes underlying these initiatives stressed planning, 
cooperation, articulation, teaching, education reform support, technology, adequate funding, and 
accountability. 



CONCLUSION 



Kentucky has moved forward rapidly on assessment and accountability and appears well on its way to 
join Tennessee in linking funding to performance. It remains to be seen how successful new 
accountability efforts will be in promoting state and institutional planning and improvement. 
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THE STATE UNIVERSITY OF NEW YORK 



Richard C. Richardson Jr. 



OVERVIEW 




|he State University of New York (SUNY) has taken the initiative in developing system-level 



performance indicators. Public reporting on how well the system is meeting established goals is 



JL. viewed as a requirement for increased budgetary and management flexibility for the system. A 
strong link exists, therefore, between planning and a public reporting system. In the system's master 
plan, the proposed indicators fall under five major planning areas: access, undergraduate education, 
graduate education and research, state needs, and management. This list includes only general indicators 
of system and institutional performance. SUNY staff anticipate that campuses will themselves develop 
more specific indicators to reflect individual institutional missions. At the time of this report, the system 
had yet to go through the first round of institutional reporting on a core set of performance indicators. 

SUNY has taken a proactive approach to developing accountability measures. Given the beginning 
stages of the process, it is unclear as yet how the results will be used to promote institutional and system 
improvement or how they will link to budget incentives. 



CHRONOLOGY AND CONTEXT 
System Design 

The nation's largest comprehensive university system, the State University System of New York 
(SUNY) is composed of 4 university centers (2 with health science centers), 13 universities, 
2 stand-alone health science centers, 3 specialized colleges, 6 two-year colleges of technology, and an 
upper-division institute of technology. SUNY also encompasses 30 community colleges and 5 statutory 
colleges, 4 at Cornell University and 1 at Alfred University. 

A board of regents oversees all public higher education in New York as well as the state education 
department. The board of trustees of the SUNY system, in turn, governs all public higher education 
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institutions in the state except for those within the City University of New York (CUNY) system. 
SUNY's trustees directly administer the state-operated campuses but with substantial authority delegated 
to campus advisory bodies. The trustees practice what is termed "general supervision" of the statutory 
colleges at Cornell and Alfred and have shared responsibility for the community colleges. 



Student Demographics 

Almost 40 percent of state students enroll at a SUNY campus. In fall 1990, the total SUNY enrollment 
of 403,028 included 192,651 enrolled in community colleges and 210,377 (167,461 full-time-equivalent 
[FTE]) in the four- year sector. (For purposes of comparison, CUNY enrolled a total of 200,700 students 
for the same period.) The student body is diverse in age, economic and social background, and academic 
preparation. In 1990, more than half of the students were women and 13 percent minority. 



SYSTEM EVOLUTION: INITIATIVES AND PRIORITIES 



When SUNY was created in 1948, it was in the face of substantial opposition from the private higher 
education sector and, to some degree, from the New York Board of Regents. 1 It was to be the last 
American state university. During the 1950s and 1960s it grew to its present configuration by collecting 
together existing state teachers' colleges, two-year institutes of agriculture and technical studies, new 
locally sponsored and operated community colleges, and three once-private institutions. Some aspects of 
SUNY's development were retarded by the persistent belief in the state of New York that the purpose of 
public institutions should be more limited than that of their private counterparts, 2 SUN Y's history can be 
summarized around two themes: (1) planning, and (2) working for the autonomy necessary to achieve 
system goals. 



Planning 

Since 1956, the board of trustees has submitted a master plan to the regents and governor every four 
years. In accordance with the master planning statute adopted in 1961 and revised in 1971, the trustees 
also provide biennial revisions and progress reports as well as individual amendments to the master 
plan. Early plans concentrated on new campuses and expanded missions. In the 1970s, the emphasis 
shifted to completing campuses and improving governance arrangements and campus life. A concern 
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with program quality and serving a more diverse student population characterized the 1980s, along with 
renewed attention to SUNY's role in graduate programs and research. 

Striving for Institutional Autonomy 

Despite earlier efforts to increase system autonomy, the Independent Commission on the Future of the 
State University concluded in 1985 that SUNY was the most overregulated university in the country. 
The governor and legislature responded to commission recommendations by granting the university 
broad authority over budget execution and reallocation of resources, significantly stronger personnel 
authority, and more control over purchasing and contracts. 

In 1993, SUNY developed policies to enhance system flexibility by allowing universities to retain their 
tuition incomes and to carry funds forward from one year to the next. Because central 
administrators believe that increased autonomy and accountability are interrelated, they coupled 
flexibility with the development of system-effectiveness indicators. While SUNY is not currently under 
direct gubernatorial or legislative pressure to develop indicators, central administrators believe that 
doing so will allow the system to be proactive rather than reactive. 3 

Current System Priorities 

The current master plan, SUNY 2000: A Vision for the New Century, outlines five overarching, system 
goals to which each campus is expected to respond in developing its own vision of blending access and 
excellence. The goals most central to this case study, access and undergraduate education, are 
summarized in detail below. The remaining three goals in the areas of graduate education and research, 
state needs, and management are summarized in condensed form. 

Access goals involve keeping SUNY open to all, regardless of financial circumstances, race, 
ethnicity, religion, age, gender, or special needs. Achieving system goals requires attention to the 
following areas: 

• Retention, completion of degree work, and subsequent success 

• Preserving system diversity 

• Transfer of students from two- to four-year colleges 
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• Affordable tuition and fees 

• Outreach to middle and secondary schools and to students who leave without completing them 

• Expanded attention to distance learning, self-paced instruction, and credit for prior life 
experience 

• Special efforts to recruit, enroll, retain, and graduate students from underrepresented populations 

• Special efforts to attract the best and brightest and to challenge the most well-prepared 

• Additional resources to offset the cuts of the 1980s and to increase quality 

Undergraduate education goals emphasize diversity, quality of curricula, and strengthened 
commitment to teaching and learning. Responding to these system goals requires the following: 

• Standards that require student mastery of oral and written communications, quantitative analysis, 
critical reasoning, and computer literacy 

• Curricula that encourage awareness of global interdependences through study of foreign languages 
and cultures, participation in student exchanges, and collaboration 

• Curricula that develop responsible attitudes and behavior in a society that values individual liberty, 
while concurrently working to assure greater equality of opportunity for diverse cultures 

• Programs that attract and graduate students in mathematics, the sciences, engineering, and the 
technologies needed to meet the needs of New York state business and industry 

• Faculty development programs and graduate education that enhance teaching effectiveness and 
faculty evaluation, and incentive systems that reward effective teaching and the application of new 
technologies 

• Dedication to the education of the whole person 

• Comprehensive curricula and programs 

• Assessment of undergraduate student learning outcomes 
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Graduate education and research goals stress SUNY's continuing development into one of the finest 
public research universities in the country. State-need goals underscore SUNY's commitment to meeting 
work force needs, strengthening collaborative work in public education, health care, social welfare, 
environmental conservation, and promoting culture and the arts. Management goals focus on the 
efficient and effective use of public resources. 

Excellent undergraduate education appears to have been an assumption rather than an objective within 
the SUNY system. The system practice of taking excellence in undergraduate education for granted, 
however, has not gone unchallenged. Governor Cuomo's 1986 response to a board of regents' planning 
document notes the failure to identify problems or to provide (except in the most general sense) a means 
for evaluating effectiveness. The governor's comments have not gone unnoticed. A recent policy paper 
by SUNY staff argues for system and campus performance reports as a way of balancing greater 
system and campus autonomy with public accountability. 

System-Level Indicators 

The concept of performance indicators, as described in Meeting the Productivity Challenge: System and 
Campus Reports, owes its genesis primarily to the Independent Commission on the Future of the State 
University. While SUNY already had an extensive statistical database, the commission 
recommended the collection and publication of data in a single comprehensive report to aid in the 
review of system performance as it relates to state needs. As part of the reporting process, the SUNY 
Board of Trustees is to review assessment results, and its chancellor is to publish an annual summary 
report for campuses, state officials, and the general public. The annual assessment will, for the most 
part, include only data already available in a variety of existing SUNY reports. What will be new is the 
inclusion of relevant data in a single comprehensive report, 4 although, it had not yet been produced at 
the time this report was written. 

SUNY has developed criteria for the indicators to be used for periodic public reporting of 
performance. 5 These indicators must meet particular criteria: (1) they are not to diminish or distort the 
diverse missions of constituent campuses, (2) they must examine performance from a system rather than 
campus perspective, (3) they are to provide tangible evidence of i<ie degree to which the goals outlined 
in SUNY 2000 are being achieved in relation to available funding, and (4) they should stimulate 
improved performance in addition to describing current results. 

While there is broad consensus on the goals to be addressed, there is less agreement about the choice of 
the indicators themselves. The proposed list includes only general indicators that reflect a system and 
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institutional perspective, leaving it to the campuses to develop more specific indicators appropriate to 
their goals. Using existing data to avoid the development of costly new information systems 
necessarily places constraints on what can be used. That decision was made, however, to permit current 
results to be compared with past performance and to ma^e individual campus data comparable with 
those of other SUNY units with similar missions. In addition, comparison was sought with the data of 
peer institutions, the system as a whole, and a composite of those of peers. 

Proposed indicators are divided into five main sections corresponding to the five general goals 
previously described in SUNY 2000. In addition to these measures, there are also indicators of the level 
of fiscal support. Proposed indicators include access, undergraduate education, graduate education and 
research, state needs, and management. 

Access 

• Enrollments by race/ethnicity 

• Transfer rates of students from two- to four-year institution 

• Trends in cost of attendance 

• Degree and certificate completion rates and time-to-degree statistics 

Undergraduate Education 

• Percent of campuses with approved plans for assessing student performance 

• Evaluation of basic skills, general and specialized education, and personal and social development 

• Pass-rates on certification examinations 

• Student perceptions of the quality of their undergraduate education as revealed in American 
College Testing surveys 

• Results from a survey by the Higher Education Research Institute reporting faculty perceptions of 
the quality of undergraduate and graduate education as well as performance on research and 
service activities 

• Proportion of resources devoted to undergraduate instruction 

• Class size by level and program 

• Alumni satisfaction survey 

Graduate Education and Research 

• Enrollment by graduate program and level 

• Graduation rates and time-to-degree for all graduate students and by race and gender 
70 
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• Trends in the dollar volume of sponsoi xl research for all campuses and for full-time faculty at 
SUNY doctoral centers 

• Faculty survey from the Higher Education Research Institute 

• Number of national faculty and student awards 

• National ratings of doctoral programs 

State Needs 

• Number of graduates by field 

• Amount of sponsored research related to economic development, environmental studies, health 
care, public education, and social services 

Management 

• Trends in student credit hours per faculty 

• Student/faculty, student/staff and student/administrator ratios 

• Ratings of the conditions of campus facilities 

• Ethnicity and gender of faculty administrators 

• Evaluation of external fundraising 

• Faculty and student surveys of management and administrative services effectiveness 



CONCLUSION 

The development of indicators for the SUNY system appears to be a logical step in a planning process 
that has been successful in building a comprehensive system and securing sufficient autonomy for 
effective operation. In fact, continuing expansion and preservation of system autonomy is contingent on 
the use of indicators. Autonomy must now be balanced by evidence of accountability for using state 
resources in pursuit of state goals. 

Underlying this logical evolution are the political overtones. For one, a long-term administration had 
occasional differences with the direction of public higher education and pointed out the absence of data 
on performance in relation to planning objectives. At least two other influences played a role, however. 
They are apparent in the board of trustees* proposal to initiate an indicator-based reporting system 
similar to those in use in other states. The first is the impetus furnished by a commission external to the 
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university. The second is the careful effort to reassure campus interests that the proposed reporting 
system does not threaten their autonomy. The SUNY effort to develop indicators appears to be carefully 
coordinated: the SUNY faculty senate, for example, has charged the senate's Undergraduate Academic 
Program and Policies Committee with responsibility for providing advice on undergraduate academic 
assessment. 6 

In the SUNY case, therefore, there is clear evidence of intent to link performance indicators to the 
planning process. Increased system accountability is viewed as the responsibility that accompanies 
increased system budgetary and management flexibility. However, there appears to be no design to link 
performance results to funding other than through some general notion of accountability for previous 
allocations. Current plans call for campus results on general indicators to be sent to campus officials 
prior to the release of the system report. After the system report is published, campuses will issue their 
results both on the larger and the more specific campus indicators, along with explanatory comments. 
System and campus reports will be bound into a single volume for each year in order to track progress 
over time. 



NOTES 



1. State University of New York, A Historical Perspective: 1975 TJirough 1990 (November 1991). 

2. State University of New York, SUNY 2000: A Vision for the Neto Century (September, 1991), pp. 18-19. 

3. Telephone interview with Joseph Burke, provost SUNY System (January 11, 1993). 

4. State University of New York, Meeting the Productivity Challenge: System and Campus Performance Reports 
(December 23, 1992), pp. 5-<5. 

5. State University of New York, Performance Indicators: Campus Autonomy and System Accountability 
(September 30, 1992). 

6. State University of New York, University Faculty Senate Resolution Modifying the Charge of the 
Undergraduate Academic Program and Policies Committee (January 31, 1992). 
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SOUTH CAROLINA 



Sandra S. Ruppert 



OVERVIEW 




s with most other case-study states, South Carolina's initial assessment efforts in the 1980s 
were institution-based, with the primary goal of improving student learning. AccountabiliT 



-A- A^was also a stated goal, but institutions were urged to develop their own reporting formats and 
compare their own data over time. Annual institutional-effectiveness reports, prepared by the 
coordinating board, summarized individual institutional accomplishments in meeting goals. Institutional 
comparisons were not a part of the plan. 

In 1992, the South Carolina legislature passed what is known as the Report Card legislation, requiring 
the coordinating board to report publicly specific data concerning higher education. This new law 
requires annual reports "to easily compare peer institutions in South Carolina and other SREB 
states with the state's postsecondary institutions." Much of the same information was 
previously collected for institutional-effectiveness reports, but the manner in which it is reported has 
changed. Information is to be arrayed in tables and matrices to enable interinstitutional comparisons. 
Further, the law requires the eventual reporting of results of a statewide alumni satisfaction survey. 



CHRONOLOGY AND CONTEXT 
System Design and Evolution 

The South Carolina Commission of Higher Education (CHE) was established in 1967 as the higher 
education coordinating agency for the state's 12 public four- year and 21 public two-year colleges. In 
addition to 33 public campuses, the state has 28 private colleges: 23 four-year institutions and 5 
two-year colleges. CHE has statutory responsibility for reviewing individual institutional budgets and 
making recommendations for a consolidated budget for both the two- and four-year colleges and 
universities. Also, CHE conducts studies and makes policy recommendations to the governor and 
general assembly, conducts program reviews, and has approval authority for new and existing programs. 
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Commission membership was restructured in 1978 and again in 1988 and is currently composed of 18 
individuals appointed by the governor to four-year terms. The executive officer is appointed by and 
serves at the pleasure of the commission. Nine boards each govern single institutions; the Board of 
Trustees of the University of South Carolina governs the research university, 2 senior, and 5 
two-year branches; and the State Board for Technical and Comprehensive Education governs 16 
two-year comprehensive technical colleges. 

Student Demographics 

• Over two-thirds of adults in South Carolina have at least a high school diploma; 17 percent have a 
baccalaureate degree or more. 

• In 1985, South Carolina's college-participation rate lagged behind both the southern region and the 
nation. Without a shift in this trend, the number of high school graduates entering college will 
decline by approximately 11 percent in the 1990s. 

• From 1980 to 1990, total enrollment in higher education increased by 20 percent. In 1990, total 
postsecondary enrollment was 159,302 students, two-thirds of whom attended college full time. 
Twenty-two percent of students enrolled were minorities. 

• Although higher education enrollments have been increasing, state funds for higher education 
operating expenses have declined 1 percent over the last two years. 

Chronology of Related Priorities and Initiatives 

1979: The South Carolina Master Plan for Higher Education is released. Among its nine goals is the 
mission to improve the quality of postsecondary education. 

1984: The legislature passes the Education Imprvvement Ac* s legislation to reform K-12 education. 
Among other things, this legislation raises standards for high school graduation and is believed to have 
prompted the focus on higher education in the state. 

1985-86: Financed by the general assembly, CHE commissions a study to evaluate higher education 
policy issues in South Carolina. 
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1986: A commissioned study by Augenblick, Van de Water and Associates (AVA) is issued. Among its 
recommendations are the call to make assessment of higher education quality a major objective in 
coming years and for CHE to increase its visibility among policymakers and citizens in state. 

May 1987: A new commissioner for higher education, Fred R. Sheheen, is appointed. A so-called 
Cutting Edge proposal, developed subsequent to the AVA study, is updated, revised, and submitted to the 
general assembly. 

June 1988: Legislative Act 629, known as the Cutting Edge legislation, passes. The act includes a 
provision for "improving accountability through planning and assessment." 

1988: With initial financial support provided by the Fund for the Improvement of Post-Secondary 
Education (FIPSE), CHE, and Winthrop University (then College), the institution-based South Carolina 
Higher Education Assessment (SCHEA) Network is formed to help colleges and universities address 
assessment issues. (Currently, the SCHEA Network comprises 40 institutions and representatives from 
the CHE and the State Board for Technical and Comprehensive Education.) 

February 1989: CHE approves Guidelines for Institutional Effectiveness, which provide colleges and 
universities with a list of 18 components of their academic and administrative operations that require 
assessment per Act 629. The institution-based process focuses on "improvement rather than on data 
collection/' Annual reporting is required, with a schedule that phases in implementation over four years. 

1990: Tne first Summary Report on Institutional Effectiveness is published, reporting on 3 of the 18 
components for academic year 1988-89. The report purposely is designed "to discourage 
interinstitutional comparisons." Its purpose is described as a means "to strengthen the quality of 
individual colleges and universities through a continuous cycle of improvement." 

1991: The first statewide strategic plan, Choosing South Carolina's Future: A Plan for Higher Education 
in the 1990s, is published. This plan is developed pursuant to Act 629. Recognizing that "the state could 
not move forward in all directions at once," CHE and the stated colleges and universities choose to 
focus on three issues: enrollment, quality, and collaboration. 

1991: The second annual Summary Report on Institutional Effectiveness is issued in response to the 
requirements of Act 629. This report describes five effectiveness components from academic year 
1989-90. 

1992: The third annual Summary Report on Institutional Effectiveness describes nine of the 
components. 
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February 1992: The general assembly passes Act 255, a statute that requires CHE to report specific data 
concerning higher education to the governor and the general assembly prior to January 15th of each year, 
commencing in 1993. 

January 1993: Combined reports on Act 255 of 1992 and the Summary Report on Institutional 
Effectiveness are submitted and the 18 effectiveness components implemented. 



ASSESSMENT OF INSTITUTIONAL EFFECTIVENESS 



In South Carolina, as in many other states, the impetus for improving postsecondary education grew out 
of the elementary and secondary education reform effort. In 1984, South Carolina passed the Education 
Improvement Act, a critical piece of legislation that, among other things, raised standards for high school 
graduation. This act is believed to have prompted the focus on higher education in the state. 

In 1985, the South Carolina legislature commissioned a study to evaluate higher education policy issues. 
The resulting report, Higher Education in South Carolina; An Agenda for the Future by Augenblick, 
Van de Water and Associates (AVA) was published in February 1986 and contains 9 
recommendations to improve quality and 13 recommendations to strengthen the commission. The AVA 
report echoed many of the national and regional studies of the mid-1980s that called for renewed 
emphasis on quality and accountability in higher education. 

Following the AVA report, six advisory task forces comprised of institutional representatives, system 
representatives, legislators, a lay citizen, and CHE staff considered the AVA findings and 
recommendations. Following task force deliberations and the appointment of a new commissioner in 
May 1987, CHE issued The Cutting Edge: Higher Education's Initiatives for Research and Academic 
Excellence in South Carolina. In June 1988, the governor signed into law Act 629, known as the Cutting 
Edge legislation. CHE described this as "a major piece of legislation to strengthen quality and improve 
accountability in higher education/' 

The Cutting Edge legislation contained new initiatives to address the preparation, admission, and 
retention of students; the improvement of instruction and research, and the recognition of excellence in 
these areas; and the strengthening of planning and quality assessment for greater accountability. 
Specifically, Part C of Article IV, "Improving Accountability Through Planning and Assessment," 
required CHE to ensure that a system for measuring institutional effectiveness was in place at every 
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public college and university. Part D required institutions to establish their own procedures and 
programs to measure and quantify student achievement. 

Process Required 

Although in accordance with CHE requirements, the Cutting Edge legislation required that all 
"state-supported institutions of higher learning shall establish their own procedures -and programs to 
measure student achievement; 1 it also specified that the program include faculty involvement in the 
process, tracking of student progress through the curriculum, and follow-up of graduates. A specific goal 
for the program was to "provide data ... that can be used to initiate curriculum, programmatic, or policy 
changes within the institution/ 1 

Assessment Domains 

CHE's Guidelines for Institutional Effectiveness, developed with input from the SCHEA Network, 
outlined the 18 domains on which institutions would report annually to the CHE. As part of their report, 
colleges and universities were expected to describe progress "in developing their assessment programs 
and to submit concrete, non-anecdotal, and quantifiable information on student achievement to the 
Commission on Higher Education." 

The 18 domains required as a result of the Cutting Edge legislation were as follows: 

1 . Student command of core requirements in general education 

2. Student command of basic knowledge in major discipline 

3. Performance of professional-program graduates on licensing and certification exams 

4. Reports of program changes resulting from external program evaluations 

5. Alumni follow-up studies 

6. Entry-level skills necessary for college work 

7. Success of entering students in meeting college or university admission prerequisites 

8. Remedial and developmental programs 

9. Achievement of students transferring from two- to four-year institutions 

10. Analysis of undergraduate retention and attrition 

1 1 . Minority student and faculty access and equity 

12. Academic performance of student athletes 

13. Assessment procedures for student development 
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14. Assessment of library usage and collection-development procedures 

15. Assessment of administrative and financial processes and performance 

16. Assessment of facilities 

17. Assessment of public service 

18. Assessment of research 



STATE-LEVEL INDICATORS 

Act 255, South CardHna's 1992 higher education report card, required that CHE report data or 
information relating to the state's public, postsecondary institutions to the governor and the general 
assembly in a number of new areas. Originally introduced by the chair of the South Carolina Senate 
Education Committee, its design was based on New Mexico's legislation. While interinstitutional 
comparison was not part of the plan in implementing the Cutting Edge legislation, the language of Act 
255 specifically stated that annual reports must be presented so as "to easily compare peer institutions in 
South Carolina and other SREB states with the state's public post-secondary institutions." 

Act 255 required that CHE "develop and adopt a format for the report and shall insure consistent 
reporting and collecting of the data in the report by the institutions." A committee made up of 
representatives of SCHEA Network and CHE staff developed definitions and agreed on a fomiat for 
reporting, which was supported by institutional presidents and approved by the education committees of 
the general assembly and by CHE, pursuant to Act 255. 

Process Required 

Some of the 10 elements for four-year colleges and the 7 elements for two-year colleges required by Act 
255 had traditionally been included in annual institutional effectiveness reports. However, under the new 
act, CHE was to array the information in tables and matrices to allow easy comparison of data from the 
three research universities, the comprehensive teaching universities and colleges, and the two-year 
institutions. Comparisons with other SREB states, where available, were also to be included. The 
narrative that was part of previous effectiveness reports was still to be used to describe the significance 
of the data. In addition to required data, the act defined the nature of a statewide alumni satisfaction 
survey. The act stipulated that "survey instruments must address the issues of overall satisfaction, 
satisfaction with major instruction, impact of general education, and current societal 
participation of alumni/' 
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Assessment domains 



Under Act 255, each four-year postsecondary institution is required to report the following data to the 
CHE (two-year college requirements appear with an asterisk): 

1. Number and percentage of accredited programs and programs eligible for accreditation (*same 
for two-year colleges) 

2. Number and percentage of undergraduate and graduate students who complete their degree 
program (*two-year colleges report on undergraduates only) 

3. Percent of lower division instructional courses taught by full-time faculty, part-time faculty, and 
graduate assistants <*same for two-year colleges) 

4. Percent and number of students enrolled in remedial courses and the number of students exiting 
remedial courses and successfully completing entry-level curriculum courses 

5. Percent of graduate and upper-division undergraduate students participating in sponsored 
research programs 

6. Placement data on graduates (*same for two-year colleges) 

7. Percent change in enrollment rate of students from minority groups and the change in total 
number of minority students enrolled over the past five years (*same for two-year colleges) 

8. Percent of graduate students who receive undergraduate degrees at the institution, within the 
state, within the United States, and from other nations 

9. Number of full-time students who have transferred from a two-year, postsecondary institution 
and the number of full-time students who have transferred to a two-year, postsecondary 
institution (*two-year colleges report on transfer to and from four-year institutions) 

10. Student scores on professional examinations, with detailed information on state and national 
means, passing scores over time, and the number of students taking each exam 

11. Appropriate information relating to each institution's role and mission (*same for two-year 
colleges) 
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REVIEW AND REPORTING OF RESULTS 



Comparative Data Use 

CHE understood that the long-term results of the Cutting Edge legislation mandated reporting to be 
primarily for improvement purposes and only secondarily for accountability. Under the Guidelines for 
Institutional Effectiveness, institutions were urged to "develop reporting formats that compare their own 
data over time and that include narrative explanations of how findings are being fed back into the 
planning process to bring about positive change or continued achievement." CHE specifically stated 
that comparing institutions was not part of the plan. 

The Summary Report on Institutional Effectiveness, published annually in 1990, 1991, and 1992 
summarized institutional information in a few pages. The CHE report placed strong emphasis on how 
colleges and universities use assessment information to address problems. Consistent with Cutting Edge 
requirements, it attempted to focus on an institution's accomplishments and concerns within the context 
of its individual goals and clientele. In fact, a number of institutions implemented major changes to 
improve programs and services as a result of this process. In addition, CHE studied the reports and 
issued separate recommendations. 

While the Cutting Edge legislation carefully and deliberately avoided interinstitutional comparisons, Act 
255 of 1992 specifically required institutional comparability. As it was now required by statute, a format 
was developed to allow for easy comparison. CHE included introductory paragraphs and footnotes 
in order to provide clarity and avoid misrepresentation. In January 1993, CHE provided to the governor 
and the general assembly the consolidated Reports on Act 255 of 1992 and Summary Report on 
Institutional Effectiveness. 

Use of Quantitative and Qualitative Data 

These reports include both qualitative and quantitative data. The summaries for reporting use a 
consistent format for each institution and the information is, for the most part, qualitative. The data 
reported under Act 255, by contrast, is nearly all quantitative and displayed in tables. 

Use of Contextual Data 

Cutting Edge legislation specifically recognized the diversity of South Carolina's colleges and 
universities and the tradition of institutional autonomy. For purposes of measuring institutional 
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effectiveness, institutions were directed to design procedures and programs "consistent with each 
institution's mission and educational objectives." Act 255, however, makes no such mention of 
institutional diversity and autonomy. By publishing a consolidated report in 1993, CHE sought to 
provide the quantitative, interinstitutional comparative data required under Act 255, in conjunction with 
the contextual, institutional-specific data required by the Cutting Edge legislation. 



CONNECTION TO OTHER PROCESSES 
Linkages to Budget 

As part of CHE's first statewide strategic plan to augment its Institutional Effectiveness Program, it 
recommended a strategy for modifying the appropriations formula in order to address quality incentives 
and accountability. CHE sought a process that could be built into the formula and thereby establish a 
level of formula funding at which incentives could work. 

It is unclear how the data generated pursuant to Act 255 will affect higher education funding 
priorities in the state. According to Act 255, CHE "shall make no funding recommendation, capital 
outlay recommendation, distribution or certification on behalf of any public, postsecondary institution 
that has not submitted the information required. . . 

Linkages to Planning 

Cutting Edge legislation directed CHE to "maintain a statewide planning system to address strategic 
issues in public and private higher education.'* Four actions were called for: 

• Identification of future directions and appropriate methods for meeting challenges 

• Review of major goals identified by colleges and universities to ascertain their relationship to 
higher education 

• Maintenance and development of quality 

• Maintenance and continued provision of access to and equality of educational opportunity 

The act also directed CHE to set an agenda for annual planning. It required that an advisory council be 
established to assist with planning. The act allowed public institutions to design and maintain their own 
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planning process but required CHE to ensure that the institutions did, in fact; maintain a planning 
process. 

Choosing South Carolina's Future: A Plan for Higher Education in the 1990s (July 1991) was the first 
strategic statewide plan. The principles on which the plan rested were ensuring access, ensuring student 
retention, ensuring real student growth and achievement, and making the higher education system 
and its institutions more accountable to the public. Based on those principles, four areas of priority 
for planning over the next decade were established: 

• Developing partnerships 

• Expanding participation and achievement 

• Improving quality and accountability 

• Establishing a continuous, shared process for higher education 

In addition, accountability was built into the plan's structure through the inclusion of key indicators of 
progress in implementing this plan. It was structured to permit annual updating. 



CONCLUSION 

As the South Carolina commissioner reported, initial institutional response was "fairly hostile'* to the 
effectiveness program required by Act 629, the Cutting Edge legislation. However, institutional 
concerns were tempered as campuses came to understand, through the SCHEA Network, both the high 
degree of control they would have over their assessment processes and the clear emphasis of the process 
on improvement rather than accountability. 

The passage of Act 255, specifically addressing institutional accountability, caused a new wave of 
concern among the institutions. The SCHEA Network, however, has been able to assume considerable 
responsibility for developing plans for implementing statewide initiatives. 
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TENNESSEE 



Peter Z Ewell 



OVERVIEW 

Tennessee provides a particularly rich case setting for the evolution of state policy on 
postsecondary educational effectiveness. Its "performance funding" approach — established / 
formally in 1979 and operated experimentally for a significant prior period — was, for many ^ j{ 
years, the sole example of such a system in the nation. Tennessee's experience with publicly reported | f/ 
higher education performance statistics began in 1984, far earlier than in most other states. Tennessee's 
experience is thus far less experimental allowing some long-term effects of policy to be separated from 
the inevitable disturbances caused by early implementation. Similarly, a long history not only has meant 
that many things have been tried but that far more is currently in place with respect to state-level 
statistics than in other states. 

On balance, state higher education policymakers appear to have been more successful in developing and 
reporting state-level indicators and in using them effectively to account for and promote the state's 
higher education system than they have been in inducing institutions to engage in their own local 
planning and assessment exercises. 



CHRONOLOGY AND CONTEXT 
System Design and Evolution 

The Tennessee system of public higher education has been unusually stable with respect to structure and 
governance. The Tennessee Higher Education Coordinating Board (THEC) was created in 1967 as a 
coordinating body and charged with a traditional array of responsibilities, including statewide master 
planning, institutional budget review and recommendation, and program coordination that includes the 
approval of new academic programs and approval and licensing authority for all postsecondary 
institutions operating in the state. Governing authority is vested in two public boards: the Trustees of the 
University of lennessee, responsible for 4 senior campuses, and the Tennessee Board of Regents 
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responsible for the balance of 20 four-year and two-year campuses, as well as 26 
non-degree-granting vocational-technical schools. Although the spans of control of these two boards 
appear uneven, each has historically been responsible for approximately half the state's higher education 
budget. These governing arrangements have been in place for over 20 years, creating an unusually stable 
climate for policy evolution. Board leadership, academic staffs and key higher education legislators 
have also enjoyed relatively long tenures. This has enabled greater than usual opportunity for 
institutions and state leaders to work out practical solutions to typical problems of policy 
implementation. 

Though funding levels for higher education in terms of dollars appropriated per full-time-equivalent 
(FTE) student have been comparatively low nationally, they have generally compared favorably with 
those of other southeastern states. Over the past two fiscal years, for example, higher education funds 
grew a substantial 18 percent; as the state's economy recovers, this commitment is expected to continue. 



Student Demographics 

The state*s higher education system as a whole enrolls about 160,000 undergraduate FTE students, with 
a student distribution approximately at national averages with respect to average load and choice of 
program. The proportion of minority enrollments, however, is somewhat higher than in other states, and 
state policies have been unusually sensitive to monitoring and improving their success. In recent years 
(1989 to 1992), enrollments have grown markedly — particularly in the two-year allege sector — with 
consequent budgetary pressures associated with serving more students with a stable budget. 



Chronology of Related Priorities and Initiatives 

Tennessee's initiatives with respect to accountability can be best summarized in terms of two related 
tracks: (1) the evolution of performance funding, and (2) the development of successive generations of 
statewide standard statistical indicators. Key dates are as follows: 

1976: Performance funding is initiated, using non-appropriated funds to create an incentive pool of 
approximately 2 percent of total instructional budget to be allocated to institutions on the basis of five 
defined performance criteria. 
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1979: Formal performance funding criteria (entitled the Instructional Evaluation Schedule) are 
formally adopted, using appropriated funds; the pool remains at 2 percent. 

1983: New, more extensive criteria for performance funding are put into place; the pool of available 
funds for allocation is increased to 5 percent. 

1984: The legislature enacts a comprehensive reform package (TCA 49-1-302), directed primarily at 
K- 12 education. The law' mandates a joint report each year by THEC and the State Board of Education 
on "the extent to which goals are being fulfilled and needs are being met" and provides the basic 
framework for THEC annual statistical reporting for the period thereafter. 

The legislature enacts a set of 19 statistical benchmarks for reporting progress in higher education (the 
so-called Bragg Marks, named for their sponsor Representative John Bragg) as part of larger reform 
effort. These benchmarks are seen as an "accountability trade-ofT to obtain new funds for education 
through an increase in state sales tax. 

1987: THEC issues a master plan and, for the first time, is able to report comprehensive statistics on 
statewide outcomes and institutional performance compiled through the Instructional Evaluation 
Schedule. The plan includes 5 comprehensive goals and 15 statistical targets for achievement. 

1989: The legislature enacts Tennessee Challenge 2000, modeled on an SREB report, in turn inspired by 
the National Education Goals process. The legislature establishes statewide goals and statistical 
targets to be used as the basis for public reporting of higher education outcomes through 1992. 

1990: THEC issues a new strategic plan consistent with the statewide plan, Tennessee Challenge 2000, 
but also incorporating 1987 master plan goals in a new form and involving lay board membership on the 
planning commission for the first time. Six themes for development are identified with quantitative 
targets for each. 

1991: The legislature enacts new requirement for public institution-level reporting of benchmark 
statistical performance indicators (TCA 4-7-210); these are largely the same indicators as previously 
reported in aggregate by THEC, obtained through Tennessee 2000 Challenge and/or the Instructional 
Evaluation Schedule. 

New performance funding standards are adopted by THEC for operation through 1996. The 
standards are considerably more quantitative and outcomes-oriented than in previous versions. 
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ASSESSMENT OF INSTITUTIONAL EFFECTIVENESS 



State Strategy and Policy Evolution 

Tennessee's approach to state policy has evolved primarily tlirough the incentives and requirements 
embedded in performance funding. Unlike the indirect approach taken by other states, from the outset 
the requirements of performance funding involved a set of common insUtuUonal performance criteria. 
But a guiding principle of the program was always that the applicaUon of these criteria should stimulate 
local institutional improvement efforts in at least two ways: 

First, the existence of statewide standards (and the incenUves associated with them) was expected to 
greatly enhance insutuUon-level assessment capability. In its original form, the InstrucUonal Evaluation 
Schedule allowed institutions substantial credit for efforts to develop new assessment approaches — 
primarily in the major field, where few instruments were then available, and in the area of student and 
alumni satisfaction. The former area was particularly fruitful for insututions. At the University of 
Tennessee, Knoxville (UTK), for example, more than a third of the academic departments needed to 
develop sound local assessment instruments, and the process of goal development that this entailed often 
affected the curriculum. A parallel effect was the creation of an assessment office that became 
recognized for its pioneering work in assessing insututional effectiveness. Though less visible than at 
UTK, enhancements in assessment capability occurred at most campuses across the state. 

Second, state performance standards were expected to directly sumulate instructional improvement in 
much the same manner as mandated targets for K-12 educauon challenged improvement in this area. 
The first two iterations of the Instructional Evaluation Schedule directly embodied the notion of 
inducing improvement by including a criterion that gave insUtutions credit for the use of assessment data 
in such areas as institutional program review, planning, and budgeUng. This mechanism caused a pattern 
of institutional reaction similar to those in other states that had more indirect, institution-centered 
assessment mandates: mos' institutions were able to report progress in establishing utilization and 
improvement processes, but few had any actual improvements in outcomes to report. 

To help foster local assessment utilization, die commission was careful not to release institutional results 
in comparauve form, though the press generally was able to obtain them directly from insututions that 
did well. Thus, from the beginning, die public tone set was less one of "accountability" than'of 
"improvement." Moreover, on many criteria, insUtuUonal results were net judged in terms of a common 
norm, but rather in terms of peer-group comparisons with similar institudons across the country (for 
example, the ACT-COMP examination) or. more interestingly, in terms of the improvement that an 
institution was able to register on a given indicator over time. 

86 

9 4 



Process Required 

The performance funding process — essentially unchanged since 1976 — requires each institution to 
submit data to THEC on a defined set of performance criteria. Each criterion is assigned a certain num- 
ber of points, with detailed scoring guides provided on how points should be assigned. The number of 
points obtained results in a specified number of dollars awarded, up to the maximum percentage of 
instructional budget (2 to 5 percent) in effect at that time. 

Assessment Domains 

In the first version of the Instructional Evaluation Schedule guidelines, institutions were assessed and 
rewarded according to five criteria: 

1. Performance in general education as assessed by the nationally-normed ACT-COMP 
examination 

2. Performance in the major field as assessed by a nationally normed discipline or professional 
examination if available, or, if not available, by means of a locally-designed examination 

3. Alumni satisfaction as assessed by survey 

4. The proportion of eligible programs in the institution's inventory that are accredited 

5. The use of assessment data in institutional planning and improvement 

As noted, these were first assessed in process terms because few actual outcomes data were avail- 
able. The latest version of the guidelines is far more prescriptive and outcomes oriented. Institutions are 
now assessed in terms of 10 standards: 

1. Objective measurement of general education outcomes as assessed by means of the nationally 
normed ACT-COMP or College-Base examinations 

2. Performance on major field examinations as assessed by available nationally normed 
instruments with performance compared to national norms 
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3. Alumni and enrolled-student surveys, with standard questions used by all institutions on student 
satisfaction and with performance compared to survey norms and/or past institutional 
performance 

4. Accreditation of accreditable programs (as above) 

5. ■ Peer review of non-accreditable programs 

6. Masters level program review results for universities; job placement rates for two-year 
institutions 

7. Institutional performance in meeting minority enrollment goals 

8. Institutional performance in retaining and graduating students against campus goals on these 
measures 

9. Institutional performance in attaining identified "planning benchmarks" established through 
campus planning processes (but visibly related to the THEC master plan or to Tennessee 
Challenge 2000) 

1(1 Actions taken at the institution to use the resulting data to make improvements 

Examination of these domains and their evolution shows them to be quite consistent with the reporting 
requirements for institutions in states with more decentralized assessment mandates. Procedures for 
compiling statistics, however, are by the nature of performance funding far more precise and 
prescriptive. This evolution is partly revealed by the size of the guidelines document itself: current 
guidelines are more than three times as extensive and detailed as the original document governing 
performance funding in 1979. 



STATE-LEVEL INDICATORS 
State Strategy and Policy Evolution 

The Instructional Evaluation Schedule was originally designed to serve a dual role. It was intended, on 
the one hand, to stimulate institutions to make improvements in both assessment practices and 
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instructional performance. On the other, it was supposed to permit THEC to collect consistent statewide 
data that could be used politically to report progress and to justify need. 

In general, the program's success has been far greater in the second of these areas. While institutional 
responses to the stimulus of performance were mixed, THEC quickly was able to use the data that 
resulted from the program in budget presentations and master plans from 1984 f-rward. Indeed, in its 
1987 master plan, THEC was far ahead of most other states in its ability to present coherent trend data 
about student outcomes. The commission's executive director has long maintained that this ability was a 
key factor in allowing higher education to protect its budget against the competing claims of other agen- 
cies throughout the 1980s. 

Statewide performance data included in the 1987 master plan were typical of the many reports THEC 
issued during this period. Included were data on productivity (e.g., enrollments and costs), equity (e.g., 
minority enrollments, persistence rates, and graduation rates), and quality (e.g., accreditation of 
programs and ACT-COMP results). As the Instructional Evaluation Schedule evolved and produced new 
data, new results were incorporated into statewide reports. For example, THEC's annual report for 1988 
contained data obtained from a common alumni survey for public institutions adopted the year before as 
part of the Instructional Evaluation Schedule. 

The passage of Tennessee Challenge 2000 and its associated reporting requirements both extended the 
scope of the performance data reported publicly and, because of the linkage to a visible reform effort, 
considerably enhanced its public impact. As before, mandated annual reports included data on all three 
areas of accountability — productivity, equity, and quality — though the statistical data provided was 
more extensive than in past presentations. More important, the format for presenting indicators data 
changed markedly for the Tennessee Challenge 2000 reports. Instead of data being provided in a more 
general narrative report or being included as background in the form of tables, statistical performance 
data took center-stage in the presentation. The result was a "report-card" look that highlighted trends and 
allowed easy review by both the legislature and the public. 

The final step in this evolution was the passage of TC A 49-7-2 10, which extended public reporting of 
these statistics to the institutional level. Previously, THEC had reported progress in attaining statewide 
goals in terms of statewide aggregates. The new law added little to the information burden as most of 
the required data were already being compiled. However, the shift in unit of analysis to the level of the 
institution signalled a major change of philosophy toward comparative reporting — a step that THEC, 
despite its insistence on the availability of common data on performance, had deliberately avoided up 
to that point. 
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Specific Domains Reported 

Collectively, the Instructional Evaluation Schedule, Tennessee Challenge 2000, and TCA 49-7-2 10 
embrace a wide range of data. Grouped specifically by case-study areas of interest, these include the 
following: 

• Productivity indicators, including degree completion rates, lower-division credits taught by 
different kinds of faculty, and two - to four-year transfer rates 

• Equity indicators, including minority enrollments, degrees awarded, and graduation rates 

• Quality indicators, including accredited programs, remediation activities and indicators of 
remedial effectiveness, pass-rates on professional licensing examinations, ACT scores of 
entering students, ACT-COMP and major field test scores of exiting students, alumni and 
current-student satisfaction data, and job-placement rates by field of study 



REVIEWING AND REPORTING OF RESULTS 
Comparative Data Use 

As noted, public reporting of performance data has evolved as follows: (a) system-level reporting by 
THEC of occasional data in the form of master plans and annual reports since 1984, (b) formal 
system-level reporting by THEC of standard data through Tennessee Challenge 2000 since 1989, and (c) 
formal institution-level reporting of indicators in "report-card" form under TCA 49-7-210 (passed in 
1992, now in process). Only the last of these involves direct comparative institutional reporting, though 
the earlier Bragg Marks involved some measure of comparison. Indeed, THEC's philosophy in 
administering performance funding and in the Tennessee Challenge 2000 reports has been markedly 
noncomparative. Performance funding uses normed comparisons to award points, but these are 
generally against established peer institutions and against the institution's own past performance. 
Tennessee Challenge 2000 used aggregate data only in the first report, but the two most recent reports 
have provided institution-level data in the appendix. From the start, THEC had the potential to supply 
such institution-by-institution data. 
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Use of Quantitative/Qualitative Data 



Virtually all of Tennessee's many indicators have been quantitative, and there has been a marked trend 
toward greater quantification over time. Early versions of the Instructional Evaluation Schedule involved 
heavy use of process indicators (such as "use of informaUon to make improvements") that involved 
considerable judgment on the pan of THEC staff, but these have evolved considerably toward more 
quantitative data, largely because such data are easier to apply fairly in awarding funds. 



Use of Contextual Data 



THEC public reports have contained considerable commentary about the data provided, though the data 
themselves are clearly visible. Tennessee Challenge 2000 reports, for instance, highlight the results 
obtained on each indicator but embed them in a brief textual analysis of trends and conclusions. 



CONNECTION TO OTHER PROCESSES 



In general, THEC iniUaUves in performance indicators have been linked to both budgeting and statewide 
master planning. This has been far less true for legislative mandates, including Tennessee Challenge 
2000 and/TCA 49-7-2 10. 



Linkages to Budget 

Established in 1976, performance funding, is a direct part of the budgeUng process, though at the current 
5.45 percent, only a small pan. More important, THEC has included trend data on ACT COMP, major 
field exam performance, and more recently, student and alumni satisfacUon-level, as an integral part of 
the budget request for higher educaUon since about 1985. In the words of THEC's executive director, 
performance funding is "the 5 percent of the budget that allows us to sell the other 95 percent." As 
additional opportunities to use performance data to make the case for higher education's budget have 
arisen through master planning and Tennessee Challenge 2000, performance data have also been used. 
The original Tennessee Challenge 2000 reporting plan prepared by THEC, for instance, set specific 
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performance targets and accompanied these with the estimated dollar amounts required to meet 
established goals. 

The application of data to budgeting has been strengthened during the past five years as benchmarks and 
standards have become increasingly rigorous. Among the identified results have been a level of alumni 
satisfaction above national averages, a job placement rate for students of 87 percent, and measurable 
gains in student learning. As institutions were able to demonstrate these improvements in quality, their 
budgets grew commensurably. 

For the most part, however, performance data have been used to make a general budgetary case rather 
than being more fully analyzed and presented to identify particular areas of future-funding needs. In this 
respect, they appear far more important forTHEC as a symbol of responsible and accountable 
management than as a source of real planning information. 

Linkages to Planning 

As noted, both the 1987 master plan and the 1990 strategic plan prepared by THEC use established 
performance indicators and link them directly to state goals. In the 1990 plan, for instance, six goals are 
noted: serving students, serving K-12 schools, economic development, quality initiatives (e.g., 
instructional innovation and rewards for effective teaching), institutional efficiency, and using 
technology to serve the needs of the state. Each of these is backed by a set of objectives, most of ■ vhich 
contain explicit statistical performance targets based on one or more of the available indicators. As in the 
case of budgeting, however, the use of data is largely accountability-oriented in its proposal to keep 
track of goal attainment. 



CONCLUSION 

Because of its long history of data collection through performance funding, Tennessee appears to have 
experienced fewer implementation difficulties in establishing performance indicators than have other 
states. Initiatives such as Tennessee Challenge 2000 and the much more institutionally-threatening TCA 
49-7-210 have gone more smoothly than have parallel report card initiatives elsewhere, partly because of 
the prior encounter with Bragg Marks in 1984-89 and partly because of familiarity with comp Y\ng data 
around the Instructional Evaluation Schedule. As noted, long staff tenures and a relatively stable 
governance context have allowed informal resolution of many reporting and public presentation 
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difficulties. At the same time, the press in Tennessee is now far less sensationalist in its reporting of 
results because the process itself has lost its novelty. Statistical performance reporting in Tennessee, in 
short, has become simply an accepted part of doing business. 

That said, the state has experienced some interesting implementation dilemmas. One is the difficulty of 
using direct performance incentives as a means to induce institutional improvement, as embodied in the 
evolution of performance funding. So long as dollar rewards to institutions are tied directly to 
statistical criteria, the need for fairness in the allocation process drives these criteria inexorably toward 
quantitative measures; at the same time, it renders their technical construction increasingly complex 
once statistical loopholes are removed. This process appears inevitable in any indicator system that 
involves high stakes for institutions. 

Because the objective is quality improvement, published statistical data have been used to report 
progress rather than to identify need. That is, performance funding has been designed to provide 
additional resources for good performance rather then for rectifying identified weaknesses. In keeping 
with this, state-level reporting has been almost exclusively centered on documenting goal achievement 
rather than determining the needs of the state and the consequent priorities that higher education ought 
to be addressing. 



The result is a pattern of data-use that strongly resembles the assumption embedded in the K-12 
experience and in the current National Education Goals process: clear performance criteria are used to 
set targets that by their very existence induce better results at the unit level. The consequence for 
Tennessee is a policy evolution that is somewhat varies from that of other case-study states. In 
Tennessee, policy for fostering institutional effectiveness is relatively underdeveloped when compared to 
the state-level-accountability that historically preceded it. 
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TEXAS 



Richard C. Richardson, Jr. 



OVERVIEW 

Texas has moved forward in several Iiigher education initiatives, although with varying degrees 
of success. These initiatives include statewide tesdng, performance funding, and state-level 
indicators of progress on explicit higher education outcome measures. The pivotal year was 
1987, when a number of these efforts first were initiated or discussed. 

In 1987, the legislature enacted the Texas Academic Skills Program (TASP), a basic skills testing and 
remediation program for all freshmen entering public colleges or universities. The Texas Higher 
Education Coordinating Board (THECB) is required to produce reports on the results of these tests and 
the remediation program. Also in response to 1987 legislative action. THECB summarized higher 
education's initiatives and priorities in the Texas Charter for Public Higher Education. Priorities and 
goals presented in this report center on access, quality, equity, and efficiency. The charter also explicitly 
calls for incentive funding to reward institutions achieving the goals specified in the report. The proposal 
to implement incentive or "performance" funding, however, was stalled for seven years primarily due to 
Texas' tight fiscal environment. 

More recently, the 1991 legislature passed a bill mandating the development of strategic plans by all 
agencies of state government, and, in separate action, charged THECB with developing 
performance-based funding proposals for higher education. In requiring strategic plans of all public 
agencies, including THECB, university systems, and individual higher education institutions, the 
legislature mandated annual reports on specific output and outcome measures. Also in 1991, the 
legislature charged the THECB to develop performance funding propo. As for the health-related 
institutions, universities, and community colleges. 

In the 1992-93 biennium, the latest performance-based funding proposals were considered but 
ultimately not passed by the legislature. Currently, data identified in performance-based funding 
proposals are collected by the Legislative Budget Board but are not linked to higher education funding. 
THECB is now challenged to modify the performance-based funding approach for future consideration. 
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CONTEXT 



The public system of higher education in Texas includes 97 institutions that operate on more than 117 
campuses. Thirty-seven universities and 8 health-related public institutions associated with 
university systems are governed by 12 boards of regents. Technical institutes are gov ned by a board of 
regents and the 50 public community college districts are each governed by their own board of trustees. 
The coordination of higher education comes under the jurisdiction of the Texas Higher Education 
Coordinating Board (THECB), which has the following responsibilities: 

• Approval or disapproval of all new academic programs 

• Development and updating oi the five-year higher education master plan 

• Approval and monitoring of postsecondary technical, vocational educational programs and adult 
vocational education offerings 

• Authorization of public community college districts 

• Resolution of disputes involving transfer of course credit between institutions 

• Administering of the Texas Academic Skills Program (TASP), which ensures that new college 
students have the reading, writing, and mathematical skills needed for college-level work 

• Monitoring the implementation of the Texas Educational Opportunity Plan for public higher 
education, designed to improve the participation of minority students in Texas higher education 

• Collection and reporting of data on higher education 

In addition, THECB administers state student-aid programs, interprets residency tuition and fee statutes, 
administers the state's research grant programs, regulates degrees awarded by private unaccredited 
institutions, and administers other contracts or special programs authorized by the legislature. 

Student Demographics 

The combined fall enrollment for Te*\as public higher education in 1990 was more than 785,000 
students, approximately one-half of whom were enrolled in the university sector. 1 In 1980, minorities 
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represented 35 percent of the Texas population. By the year 2000, they will represent 40 percent of the 
entire population and 45 percent of the college-age population. These demographics are not reflected in 
college enrollments. African-Americans, who represent 14 percent of the college-age population, 
represent only nine percent of enrollments in the university sector. For Hispanics, the comparable figures 
are 25 percent and 12 percent. Because these figures include historically African- American and 
predominately Hispanic institutions, they substantially overstate the success of many institutions in 
changing their demographics to reflect the changing composition of the population. 



System Evolution: Initiatives and Priorities 

There have been several major modifications to the board's authority since its creation in 1965. An 
important change came in 1987 when the 70th legislature charged the board to develop and administer a 
basic skills testing program for all entering college freshmen at public institutions, administer the 
funding allocations for four major research programs, develop a statewide higher education 
telecommunications network, conduct a sunset review of all doctoral programs, and review institutional 
core-curriculum policies. The 70th Legislature also changed the name of the board and its agency to the 
Texas Higher Education Coordinating Board (THECB). 2 

THECB summarized its initiatives and priorities within the six broad principles suggested by the 7957 
Texas Charter for Public Higher Education. These principles and a representative set of associated 
activities were as follows: 

1. Access to higher education, including educational opportunity planning and related 
programmatic activities encouraging at-risk minority students to complete high school and 
consider college and administering grants and loan programs 

2. Quality in higher education, involving efforts to improve faculty salaries; to award 
competitive grants to strengthen the skills of teacners and improve the quality of instruction in 
mathematics, science, and foreign language; to preserve the quality of degree programs; to 
deveiop the Texas Academic Skills Program; and to administer a variety of research programs 

3. Diversified educational opportunity, as reflected by the approval of new degree programs and 
sunsetting vocational courses from the state's inventory; provision of state funds to help support 
the training of family practitioners to work in medically underserved areas; sunset review of 
doctoral programs; and review and approval of role and scope (mission) statements 
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4. Funding, including support for more flexible legislative appropriations to public senior 
colleges and universities and for administering trusteed funds, junior college state aid, and other 
grant programs 

5. Efficient and effective management, involving working with other agencies to address the 
improvements recommended in a study commissioned by the Select Committee on Higher 
Education; requiring all institutions to review their campus physical planning processes; and 
studying means to ensure that public community colleges operate within their legally defined 
roles and missions 

6. Leadership, as represented by a demographics conference; a special committee on 
postsecondary medical, dental, and allied health education; a study of ways to improve 
educational opportunities in South Texas; and participation in a cooperative effort to implement 
an effective regional planning process for vocational and job training resources 3 



UNDERGRADUATE EDUCATION 
Statewide Testing 

There are a number of strands to state-^vel efforts in Texas to improve undergraduate education. 
Arguably, the most important and far-reaching of these has been the effort to develop and implement a 
testing and remediation program in higher education. 4 After visiting programs in New Jersey and Florida 
and interviewing a wide-range of knowledgeable individuals, the members of a committee on testing 
advanced seven far-sighted recommendations to cope with the approximately 30 percent of freshmen 
they estimated were entering college each year in Texas without being able to read, communicate, or 
compute at levels needed to perform effectively. 

The first recommendation called for a uniform test of reading, writing, and mathematics skills for all 
freshmen entering a public college or university. (The test, however, was not to be used in the 
admissions process.) Recommendation two urged strong advising programs for appropriate course 
placement early in a student's degree program. Recommendation three required each public institution to 
make available non-degree credit remedial opportunities on its own campus to students identified by the 
test as needing assistance. Recommendation four required remediation whenever a component of the 
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basic skills test was failed. All components of the test had to be passed before the completion of 60 
semester credit hours. If this were not accomplished, further coursework would be limited to lower 
division courses until all components of the test had been passed. Recommendation five required annual 
reports from each college and university on the results and effectiveness of remediation. 

Recommendation six called for THECB to develop the test with active participation of faculty members 
from Texas colleges and universities. Recommendation seven called for financial support from the 
legislature both to develop the test and to support institutional remediation efforts. 

After the Committee's recommendations were adopted by THECB in 1986, they were enacted into law 
during the 1987 legislative session as the Texas Academic Skills Program (TASP). Although the testing 
component has received the most attention, the program also includes academic advising, placement 
into courses at appropriate levels, and remediation where necessary. To ensure that students are receiving 
quality academic support, an evaluation of advising and remedial programs is conducted to determine 
program effectiveness. 5 

The development of TASP involved several thousand faculty in identifying the necessary skills for 

success in an undergraduate degree or certificate program. The development of the program's evaluation 

component produced three principles significant to efforts to improve undergraduate education: 

(1) student academic progress would be monitored over time, (2) multiple indicators of student progress 

and performance would be developed to judge the effectiveness of remedial programs, and 

(3) performance of skill-deficient students would be compared with those who did not require remedial 

interventions. 6 

In 1992, the first annual report was published on the effectiveness of remediation for freshmen who 
entered colleges and universities during the 1989-90 academic year. While the report urged caution in 
interpreting results, it nonetheless provided a detailed listing of the results achieved by institutions along 
with descriptions of student characteristics. 

Other Efforts 

The Master Plan for Higher Education 1990 and the Board Action Plan 1 99 1-92 contain other effotts to 
improve undergraduate education. One is developing a strong undergraduate core curricula through the 
periodic review of institutional reports submitted on a five-year cycle. A second emphasizes faculty 
attention to teaching as well as attempts to enhance professional development. THECB actions to 
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implement the emphasis on faculty include encouraging more emphasis on undergraduate teaching in 
tenure and promotion decisions, monitoring the number and percent of lower- and upper-division 
courses taught by tenure-track faculty, and establishing recognition and incentive programs to promote 
excellence. 

As was the case in other states, goals to improve undergraduate education are inseparably 
intertwined with continuing attention to access. Specific board actions promoting access include 
developing partnerships with public schools to increase the number of minority and 
economically deprived students prepared to enter higher education; providing academic and career 
counseling, tutoring, role models and additional student services through educational opportunity 
programs; eliminating economic barriers; enhancing transfer between community colleges and four-year 
institutions; and exploring alternative delivery systems and flexible scheduling for an increasingly 
non-traditional student clientele. 

All these efforts have been accompanied by a continued emphasis on working with the public schools, 
improving teacher preparation programs, tracking students to identify exit and entry points, and 
mainta; ning freedom of mobility among institutions. A key piece of 1987 legislation (Senate Bill 543) 
require:; postsecondary institutions to report first-year student test scores, developmental courses 
required, and grade point average to the sending high school or community college. 

One of the most interesting trends has been the movement from input or process to output 
indicate rs. Much of the change seems attributable to an emphasis on strategic planning coming from the 
governor and the legislature, described in the next section of this report. 7 

State-Level Indicators 

Incentive funding was proposed in the 1987 Texas Charter for Public Higher Education as a means of 
rewarding institutions that achieved specified goals. Among the 14 priorities identified in the charter 
were achievement of minority recruitment and retention goals; attainment of specified graduation rates; 
impro vements in use of campus buildings; assessment of the graduate performance as a measure of 
general-education outcomes (for example, assessing performance of a sampling of graduates on an 
appropriate standardized test); and evaluation of instructional programs through a survey of students, 
recent alumni, and the community/employers. The legislature, however, did not appropriate funds for 
this early performance-funding proposal. Efforts to implement performance funding were delayed until 
the 1992-93 legislative session, primarily because of Texas* budget difficulties. 
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In 1991, the legislature took two important steps towards reviving performance-based funding. First, it 
passed House Bill 2009 (effective August 26, 1991), which mandated the development of six-year 
strategic plans by all agencies of state government. The term agency is defined to include THECB, 
university systems, and institutions of higher education. In the language of Bill 2009, "These plans were 
to establish performance measures and to identify outcomes which institutions of higher education 
should pursue." 8 

The bill specifically requires agencies to provide each of the following items: 

1. A statement of mission goals and objectives 

2. Measures of the output and outcome of an agency in terms of indicators to be developed 

3. Identification of priority and other service populations or other service measures and how these 
populations are expected to change within the period of the plan 

4. An analysis of the use of current agency resources in meeting present needs and expected future 
needs and additional resources that may be necessary to meet future needs 

5. An analysis of any likely or expected changes in the services provided by the agency due to 
changes in state or federal law 

6. Plans and strategies for meeting current and future needs and achieving the goals established for 
the particular area of state government 

7. Other information that may be required 

The administrative instructions accompanying the strategic-planning mandate place substantial emphasis 
on quantification. Objectives, for example, are defined as "clear targets for specific action, more detailed 
than goals objectives and have shorter timeframes and stated quantity. An objective is achievable, 
measurable and sets the direction for strategies." Outcomes are defined as "quantified results or impacts 
of government action. Progress is assessed by comparing outcomes to objectives through the use of 
measures . . . outcomes are not outputs; an output is the quantity of the service or good produced; an 
outcome is the result or impact of the output." 

Also in 1991, the General Appropriations Act charged THECB with developing individual 
performance-based funding proposals for health-related institutions, universities, and community 
colleges. As originally proposed by THECB, performance-based funding plans were to be add-on 
incentive programs, constituting 5 percent of institutional budgets. Plans were to emphasize outcomes 
and output measures. However, in incorporating performance-based funding proposals in the 1992-93 
general appropriations act, the legislature identified performance funding as a part of base funding, 
rather than an add-on. Furthermore, the legislative proposal suggested performance funding at 10 
percent of total higher education funding. This 1992-93 legislative action did not pass, in part, because 
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of changes to THECB's original proposals. Institutions would not support performance funding 
amounting to 10 percent of the base budget. Also changes in legislative leadership between 1991 and 
1993 affected the bill's overall support. 9 

Currently, the data identified in performance-based funding proposals are collected by the Legislative 
Budget Board but are not linked to higher education funding, THECB has now been challenged to 
modify the performance-based funding approach for future consideration. 



REPORTING PROCESS 



The process of developing and using performance indicators in Texas is still evolving, as is the reporting 
process. Clearly, the strategic plans required of THECB and public universities will constitute the cor- 
nerstone of the process. Plans, which must be updated every two years, are designed in part "to provide 
a context to link the budget and other legislative processes to priority issues, to impose continuity in 
budgeting, and to improve accountability for the use of state resources," 10 The house bill which created 
this process includes the following provision: 

(section 5) Consideration of Strategic Plans in Performance Audits. The comptroller of public 
accounts, the Sunset Commission, the state auditor, the legislative budget board and any other 
agency that conducts performance audits of any agency shall consider the degree to which the 
agency conforms to its strategic plan in its evaluation. 

Under this mandate, THECB and the universities and university systems it coordinates are more tightly 
bound to the pursuit of state priorities and to the reporting of progress in quantified ways. Many out- 
come measures specified for the system found their way into subsequent versions of the Status Report 
on Higher Education in Texas, published biennially by THECB. The January 1992 version, for example, 
makes reference to the framework of goals adopted as part of the Master Plan for Texas Higher 
Education and the five-year Equal Educational Opportunity Plan. Future reports will undoubtedly 
reflect the influence of the board's new strategic plan as will the companion Statistical Report. Even 
more interesting, perhaps, is the requirement that universities prepare comparable plans for which they 
will presumably be held accountable. 

A second important element of the reporting process is the Annual Report on the Effectiveness of 
Remediation. As noted, this report provides specific information on student characteristics and 
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performance as well as on performance of the system as a whole. The information collected for this 
report relates both to performance state-aid and to the distribution of funds the legislature appropriates 
for remediation. 

Apart from these annual reports and the information collected biennially as part of the budget 
presentation to the legislature, THECB prepares periodic status reports on issues of concern. Examples 
include the fall 1990 report on Faculty Teaching Assignments in Texas Public Universities and the 
December 1991 report, Parity 2000: Achieving Equity for Women in Higher Education, The agency's 
own performance is evaluated on a ten-year cycle by the Texas Sunset Advisory Commission. 

CONCLUSION 

The effort to shape the state's public energies and outcomes is as large and complex as Texas itself. The 
tightly designed strategic planning and budgeting process is an enterprise of almost unimaginable 
magnitude. The leadership of this undertaking, however, is not crisply defined. In Texas, the legislature 
has been identified most often as the guiding force. Yet, it is instructive to note that the legislature very 
often acts on the recommendations of THECB committees. One good example is the Texas Charter for 
Public Higher Education. 

Three primary explanations may be offered for this situation. First, the legislature and the THECB have 
worked closely to bring a politically powerful, non-system of institutions of public higher education 
into alignment with state policy, and to do so without giving up the commitment to maintain both access 
and quality. Second, the legislature has consisi^ntly found it necessary in this process to enlarge the 
scope of the coordinating board responsibilities in spite of system and institutional resistance. Third, it is 
particularly interesting that Texas found it necessary to create a strategic planning process as part of its 
effort to develop performance indicators (or vice versa). Texas had not previously been noted for 
emphasis on planning for its higher education system. The effective use of performance indicators, 
therefore, would appear to depend on the existence of a planning process that permits evaluation of 
indicators in relation to planned objectives. 
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VIRGINIA 



Peter T Ewe 11 



OVERVIEW 

Virginia's approach to assessment and academic planning is perhaps the Inost decentralized of 
the 10 case-study states. Virginia initiated state-mandated assessment earlier than most and 
quickly became known as a national model for institution-centered assessment, providing 
an alternative to the more centralized and standardized initiatives developed by Florida, Tennessee, and 
New Jersey. Widely imitated by many states between 1987 and 1990, Virginia's approach sought to fuse 
in a single process the agendas of improving instruction at the institutional level and demonstrating 
accountability on a statewide basis. The Primary emphasis of the process, however, wa? always 
improving institutional effectiveness, especially for undergraduate instruction. 

Statewide planning efforts visibly stressed this agenda and particularly emphasized the connection 
between assessment and curriculum reform. In contrast to such states as Illinois, however, there 
was no well-established tradition of state-level planning in Virginia when the state began its assessment 
effort. Linkages between assessment and improvement, therefore, were expected to be 
primarily institution-based, with general themes identified in state-level plans but in no way dictated 
by them. 

Virginia's approach typified the state approach during the 1980s, emphasizing curricular renewal and 
experimentation in the context of abundant resources. Since 1990, Virginia's climate for higher 
education has shifted markedly, resulting in an evolving planning agenda and growing doubts about the 
long-term viability of decentralization. While no concrete mandate for performance indicators is in 
place, the pressure is rising for concrete, easily-understood measures of progress, and the State Council 
of Higher Education for Virginia has begun exploring ways to assemble institutional performance 
statistics on a more consistent basis. 

While the track record of assessment in inducing local improvement has been quite positive, at times it 
has been both uncertain and uneven. Current policies on assessing and reporting effectiveness are in a 
period of flux. Given the slate's decentralized governance traditions and its substantial existing 
investments in an institution-centered approach, Virginia is unlikely to abandon its commitment to this 
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alternative. But additional mechanisms better suited to the roles of accountability and statewide planning 
are likely to emerge within the next two years. 



CHRONOLOGY AND CONTEXT 
System Design and Evolution 

The extremely decentralized Virginia system of higher education has a long tradition of institutional 
autonomy and commitment to quality. Most of the 16 four-year colleges and universities have their own 
governing boards, set their own tuition rates, and make their own academic policies. Most carefully 
cultivate distinctiveness and consciously attempt to project a "private" rather than a "public" 
institutional image. The state's 24 community colleges, which are organized as a system, are 
considerably more uniform with respect to policy and governance. All are comprehensive community 
colleges offering an approximately even balance between transfer-related and occupational programs. 

The State Council of Higher Education for Virginia (SCHEV) was created in 1954; in 1974, its 
functions were revised. The council serves as the statutory coordinating body for higher education with 
formal authority for statewide planning and program approval for public institutions. In addition, 
SCHEV prepares budget guidelines and formulas, reviews individual institutional requests, and 
recommends budget allocations to the state. 

The council is the licensing body for all private degree-granting postsecondary institutions operating in 
the state. Revenues and expenditures per full-time-equivalent (FTE) student in Virginia attained relative 
parity with national averages by 1987 but have recently eroded due to increasing fiscal difficulties. A 
noticeably Jiigher than average proportion of total revenues is obtained for Virginia public institutions 
through tuition charges. 

Student Demographics 

The state's higher education system enrolls about 225,000 FTE students, about 85 percent of whom 
attend public institutions. The average proportion of pan time attendance is approximately equivalent to 
national averages, with somewhat higher proportions of students majoring in the traditional disciplines 
of the arts and sciences than in most other state systems. 
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Chronology of Related Priorities and Initiatives 

Virginia's initiative with respect to assessment and effectiveness reporting involves several 
components. The first is mandated institutional assessment, which began in 1985-87. A second is 
regular state-level planning. Each biennium, SCHEV is required to report on the condition of higher 
education for the governor, general assembly, and the public in The Virginia Plan for Higher Education, 
this report increasingly has included assessment information. The third component is a statewide 
strategic-planning effort entitled The University of the 21st Century (U21) begun in i988; this initiative 
has been used by SCHEV as a vehicle to raise issues about restructuring and about the overall 
direction of higher education. This reporting has taken place within the context of the state's rapidly 
deteriorating fiscal condition. Funding first became a factor in the 1989-90 biennium, when it 
brought issues of productivity and restructuring to the fore in both K-12 and higher education. Key 
dates are as follows: 

1985: The Virginia Plan for Higher Education , issued by SCHEV for the 1985-86 biennium, mentions 
"assessing achievement" as a topic the state should address. 

The general assembly passes Resolution 125 calling for a study by SCHEV "to investigate means by 
which student achievement may be measured to assure the citizens of Virginia the continuing high 
quality of higher education in the Commonwealth/' 

James Madison University is granted funds to implement a demonstration project intended to pilot test 
four different approaches in a systematic manner, each modeled on a nationally prominent 
institutional assessment program. 

1986: SCHEV presents its response to Resolution 125 as Senate Document 14, recommending that 
"each institution establish an assessment program to measure student achievement" under guidelines 
established by the council and that the results be presented biennially as a part oi* the Virginia Plan for 
Higher Education. These recommendations are adopted as policy by the general assembly through 
Senate Joint Resolution 83. 

1987: In April, SCHEV consults with the institutions and develops a set of official guidelines for the 
development of campus assessment plans to be submitted for approval by the council in July; these 
address principles and processes and direct institutions to report on achievements in general education, 
the majors, the effectiveness of remediation efforts, and alumni success and satisfaction. 
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All institutions submit assessment plans as required. To provide an incentive to cooperate, SCHEV 
decides to approve proposals, ap.d at the same time, determine each institution's eligibility to receive a 
range of special-initiative funds. All plans are ultimately approved, though resubmission is required in 
two cases. Each institution also is granted additional funds to underwrite the proposed local assessment 
effort. Summaries of institutional plans for assessment, together with a summary of SCHEV 's overall 
assessment approach, constitute a major portion of the narrative for the 1987-88 biennium. 

1988: SCHEV issues the document, The University for the 21st Century, which contains a vision for 
higher education's future. Central themes are undergraduate curriculum reform and forging stronger 
linkages between higher education and society. 

1989: The Virginia Plan for the 1989-90 biennium reports the first results of assessment 
institution-by-institution. No common data are presented, but themes of implementation are summarized 
in a general narrative. 

The general assembly passes a resolution that includes funds for assessment as part of each 
institution's base budget. This regularizes the process but also allows institutions the opportunity to alter 
or reallocate these funds internally as they see fit. 

1991: SCHEV issues a report. Colleges and Universities for the 21st Century, in partnership with the 
presidents of the public universities. Citing increasingly poor fiscal conditions, this report calls for 
changes in administrative structure to increase decentralized management and control. Assessment is 
highlighted as a positive example of how institutions are changing themselves from within. This 
document is included as part of the Virginia Plan. 

The Virginia Plan for the 1991-92 biennium contains a second round of assessment results. These are 
a so presented on an institution-by-institution basis, but. increasingly they emphasize local changes 
resulting from assessment. A general narrative entitled "Assessment at the Five- Year Mark" notes 
successful examples but expresses growing concern that most institutions are "somewhere between 
results and action" and that dissimilar local assessment programs and results do not enable SCHEV to 
present a coherent overall picture of higher education's condition and accomplishments. 

1993: SCHEV issues Change and Improvement in Virginia Higher Education as a second update on 
implementation of the University of the 21st Century. The report responds to the state's poor budgetary 
condition and emphasizes that changes in the structure, management, and delivery of higher education 
are imperative. Among its recommendations is for SCHEV to begin collecting common data on 
institutional function and effectiveness to aid planning and to monitor progress for public reporting. 
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Institutions submit their third report on local assessment results as scheduled for inclusion in the Virginia 
Plan for the 1993-94 biennium 



ASSESSMENT OF INSTITUTIONAL EFFECTIVENESS 
State Strategy and Policy Evolution 

Virginia's institution-centered approach to assessment reflects both the realities of its higher education 
structure and an explicit theory of change. Strong traditions of institutional autonomy and decentralized 
governance markedly constrain SCHEV's ability to develop common approaches in any arena. Because 
of this, it was decided that requiring institutions to engage in assessment would be the most effective 
means to promote lasting attention to undergraduate teaching and learning. By providing the occasion 
for raising important but neglected cunicular questions, it was expected that assessment would induce 
institutions to improve from within. Once identified and reported, institutional actions would then 
automatically satisfy external demands for responsiveness. The same program, in short, was expected to 
result in both demonstrable accountability and institutional improvement. 

This basic strategy was a product of the policy conditions of the 1980s. Virginia colleges and 
universities were ihen enjoying unprecedented funding levels. Questions were in the air, however, 
regarding undergraduate cunicular coherence and reform, stimulated by such reports as tthe National 
Institute of Education's Involvement in Learning and the Association of American Colleges* Integrity in 
the College Curriculum. Consistent with what was then standard good practice in developing policy, 
SCHEV's approach emphasized positive incentives by underwriting much of the costs of assessment 
and by allowing institutions maximum discretion in determining what and how to assess. The council 
avoided comparisons of any kind — including the quality of institutional assessment efforts and the 
outcomes obtained — and viewed the biennial review process as much as a means to provide 
institutions with feedback on their programs as it was a mechanism for ensuring compliance. In short, 
the policy objective throughout was consistent: to create an ongoing, locally-owned process of 
continuous self-examination and improvement. 

As with all decentrafeed initiatives, the success of this approach varied considerably with each 
campus's leadership and commitment to change. A number of institutions actively resisted the initiative 
from the outset. Others saw it as a major opportunity to engage in local efforts that they themselves had 
planned but for which they lacked leverage or resources to implement. Especially in the early years of 
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the initiative, the results were very uneven across institutions. Some institutions had a great deal to 
report in terms of assessment activities, findings, and action while others did not. In addition, these 
efforts took a great deal of time to develop. The biennial Virginia Plan Update of 1993 showed a clear 
track record of progress at a majority of institutions, but this was six years after institutional initiation 
and some eight years after the policy was formally adopted. Finally, even as rich, multifaceted and 
undoubtedly useful for local purposes as they were, reported assessment results did not provide a 
coherent "accountability story." This was not very important while funds were ample and political 
support for higher education sound; but its salience grew after the 1989 budgetary shortfalls and a 
change of administration. As fiscal conditions worsened in Virginia, the central terms of the policy 
agenda also shifted. With rising demand and limited resources, restructuring rather than quality 
enhancement became the dominant theme. The council's rhetoric and planning agenda for the nineties 
J" .asingly reflected this new theme, with the role of internal assessment in the policy scheme not as 
yet clear. 

While these factors have led to growing uncertainty about the validity of decentralized assessment as an 
omnibus state policy, they have not tarnished some obvious accomplishments. Virginia's approach has 
had undoubted impacts on institutional effectiveness — perhaps at this point the best documented 
impacts in the nation. On most campuses, assessment has clearly affected and revitalized discussions 
about curriculum, undergraduate teaching, and learning in general. These discussions have occurred 
when they have been most needed — just in time to inform much harder and fiscally induced debates 
about academic priorities. The Virginia strategy seems, therefore, to be strong in enhancing institutional 
effectiveness. However, it is unclear whether emphasizing this dimension alone has been sufficient to 
achieve accountability and to inform a growing set of planning needs at the state level. 

Process Required 

The assessment process in Virginia remains governed essentially by the terms of the original SCHEV 
guidelines. Each institution is required to conduct a formal student-assessment program that meets 
specific guidelines, and to submit a report to SCHEV each biennium. This report must describe the 
structure of the program, the methods used for assessment, the principal findings and their 
implications, and any actions taken in response. 

The report must cover each of the assessment domains noted in the guidelines, including general 
education, the major field, remedial activities and success, and alumni follow-up. The submitted report 
is reviewed by SCHEV and by a team of consultants to determine if the institution is in compliance and 
to provide institutional feedback. In the first round of reporting, expert consultants drawn from outside 
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the state conducted the reviews, but in its latest rounds the bulk of the review has been under the 
auspices of a statewide committee, with a single consultant retained as an external observer. 



The primary consequences of approval are that the institution is ruled eligible to participate in a range of 
addition-to-base incentive funding programs administered by SCHEV (many of which are not directly 
related to undergraduate education). This linkage provides a substantial incentive for many institutions, 
as such programs have remained active even in tight times. In the second years of each biennium, 
institutions are retired to complete a brief questionnaire on current activities and progress in 
assessment. These are for information purposes only and are reviewed by SCHEV staff. 



Assessment Domains 

Assessment domains are essentially governed by the SCHEV guidelines established in 1987. According 
to these guidelines, institutions must report biennially on the following: 

• Student achievement in general education 

• Student achievement in the major field 

• Current standards of college-preparedness with respect to verbal and quantitative skills and the 
numbers of students assessed as deficient in each of these areas 

• Success of remediation programs directed at such students 

• Experiences of alumni with respect to further education, job placement, and overall satisfaction 

Institutions may use any appropriate method for gathering and analyzing evidence around these topics, 
and the resulting reports are indeed quite diverse. The major governing phrase in the original guidelines 
is that the resulting reports "should be concrete, more than anecdotal, and presented in quantitative 
form." 

In some areas, however, clear behavioral standards have evolved through the ongoing review and 
feedback process. In evaluating the success of developmental programs, for instance, institutions are 
expected to have implemented a mechanism to track the grades of students exposed to such programs in 
order to determine their performance in later college-level work. 
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STATE-LEVEL INDICATORS 



State Strategy and Policy Evolution 

The assessment program in Virginia was intended to discharge accountability obligations without 
producing any statewide indicators of performance. The primary vehicle used to present system-level 
accomplishments — the Virginia Plan, prepared each biennium for submission to the governor and 
general assembly — presents a set of standard statistics on statewide enrollments, faculty, and costs, but 
contains no readily accessible data about educational processes or performance. 

This is not to say that no information on these topics exists. The 1987, 1989 and 1991 Virginia Plan 
updates contain extensive state-level narratives about the assessment process. By 1989, fully half of each 
institution's narrative addressed the structure and findings of its assessment program. There also have 
been attempts to standardize these narratives by providing common headings consistent with assessment 
guidelines and, occasionally, by developing a tabular format for reporting. The 1989 Virginia Plan, 
moreover, supplemented each institution's narrative with a brief statistical profile. In addition to standard 
enrollment figures, this profile contained such items as "the most popular baccalaureate majors" and the 
percentage of instructional activity by discipline. 

Reports provided through the parallel U21 process were similar. The U21 document itself, issued in 
1988, was almost entirely narrative in nature, although it contained considerable environmental scanning 
information throughout. Both this document and the following 1991 update report also mentioned the 
assessment process and its continuing benefits in inducing curricular reform. As in the Virginia Plan 
updates, the general benefits of the assessment process are prominent in U21 reports, but these rarely 
contained any statistical results. 

While this approach undoubtedly demonstrated that things were happening at the institutional level, it 
had major shortcomings as a report of progress. Several incidents in 1991-93 highlighted this 
difficulty, including an unfavorable newspaper report on the assessment process written by a reporter 
frustrated by the inability to rank institutions by quality. Similarly, SCHEV was experiencing problems 
in selling higher education's budget to the legislature in the absence of readily digestible statistics on 
performance. Reflecting these growing difficulties, the Virginia Plan's 1991-92 narrative on assessment 
was unusually candid: "the difficulty of Virginia's approach to assessment is that it doesn't yield one big 
story, but instead many small and medium-sized ones about the effects and progress of higher 
education." 
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A 1993 report by SCHEV to the governor and legislature, issued as the second progress report on the 
implementation of U21 and entitled Change and Improvement in Virginia Higher Education, 
suggested a shift in direction. With funding and restructuring issues at the top of the agenda, this report 
included a proposal that the council collect and disseminate a common set of institutional performance 
statistics to monitor conditions in higher education and "to tell Virginians how faculty and institutions 
are affected" by these developments. 



Specific Domains Reported 

State-level reporting of assessment results through the Virginia Plan from 1987 through 1993 
continued to follow the domains established by the original SCHEV assessment guidelines of 1987. 
These included the following: 

• Outcomes of general education 

• Outcomes of the major 

• Standards for determining the need for remediation by skill area, and the proportion of those 
entering students assessed as deficient 

• Outcomes of remedial/developmental instructional programs 

• Alumni success and satisfaction. 

No common measures are used, and only narrative reports of findings provided. In addition 
institutions are expected to report publicly on several aspects of their local assessment process: 

• Assignment of responsibility for assessment and levels of faculty and administrative involvement 

• Procedures used to assess student achievement under each of the headings above 

• Timetables for implementation 

• Costs of the effort 

• Actions taken as a result 
112 
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These were presented in narrative form as well, with institutions largely determining both the form and 
the content of their individual summaries. These summaries were then included in the Virginia Plan 
together with a statement of each institution's mission and future plans. 

Proposed common indicators contained in the 1993 SCHEV Report are far more specific, and resemble 
in many ways the types u performance indicators now collected by other states. Proposed indicators 
include the following: 

• Admissions standards for first-time students and the actual scores achieved to meet these standards 

• Prior high -school courses taken, and levels of remediation required and engaged in by first-year 
students. 

• "Profiles of teaching and learning" at each institution, consisting of average class sizes, the 
proportion of undergraduate students who experience courses taught by full or associate 
professors, the proportion of undergraduate students who experience small classes or seminars, and 
the proportion of undergraduate students who graduate with a "summarizing experience" such as a 
thesis, recital, or comprehensive examination. 

• Graduation rates in four, five, six, and seven years, broken down by race and gender 

• Post-graduation profiles of recent graduating classes, including levels of enrollment in graduate 
schools and employment placement rates 

• Quality of the institutional assessment program in place at the institution 

• Amount of extramural research funds attracted by the institution 

The majority of these proposed indicators as yet have no quantitative standards, and several are 
"process" indicators; for which counts or ratios are inappropriate. Nevertheless, its specificity and 
comparative nature renders this proposal a considerable departure from traditional Virginia practice. 
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REVIEW AND REPORTING OF RESULTS 
Comparative Data Use 

The Virginia approach has historically avoided any form of direct comparison among institutions. When 
assessment guidelines were first developed in 1987, there was considerable controversy over the degree 
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to which institutions should be compared. The 1987 Virginia Plan, for instance, contained tabular 
summaries of each institution's planned assessment approach, using categories noted in the guidelines. 
Even though each summary display was presented independently — together with the institution's 
individual assessment narrative — institutions claimed that this practice violated the original 
understanding that assessment results would not be used comparatively. Since that time, summary 
displays have been explicitly avoided and only the individual narratives presented. There has been a 
continuing evolution toward the inclusion of summary statistical data, though not in the outcomes area. 
The 1989 Virginia Plan update, for instance, contained summary data on courses taught by discipline, 
and enrollments and baccalaureate degrees granted by major field for each institution. But at no point 
has a single public display been presented containing all institutions. The 1993 SCHEV proposal for 
collecting standard indicators 10 be used for monitoring would presumably violate this tradition. 

Use of Quantitative and Qualitative Data 

The vast majority of Virginia's public reporting has been qualitative in nature — both in the Virginia 
Plan updates and in reports on the implementation of U21. Individual institutional narratives on 
assessment contained each biennium in the Virginia Plan average two to three typeset pages. These 
describe assessment procedures, general findings, and implications. But despite the guideline's directive 
that results be "non-anecdotal, presented in quantitative form," these summaries rarely contain any 
actual scores or achievement levels. Results of this kind can be obtained only by referring back to each 
institution's detailed report to SCHEV and those, although publicly available, are not publicly reported. 
Similarly, SCHEV's own narrative assessment summary prepared for inclusion in the Virginia Plan 
contains thematic observations only. Proposed 1993 SCHEV monitoring-indicators also contain items 
that are qualitative in nature — for instance, "the quality of the institution's assessment program." 
Predominant reliance on qualitative reporting has resulted in considerable richness and depth, but as 
noted, provides little ability to provide external audiences with simple answers to questions about higher 
education's performance. 

Use of Contextual Data 

The nature of the mechanisms used allows considerable reporting of context. Institutional 
narratives in the Virginia Plan, for instance, contain much information on the institution's distinctive 
mission, student clientele, and academic program — all presented prior to the discussion of assessment 
and its results. Most institutions also carefully explain any actual assessment results in light of their own 
distinctive conditions and situations. 
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CONNECTION TO OTHER PROCESSES 



Linkages to Budget 

Virginia's assessment program has been linked to other state-level initiatives and is part of an overall 
SCHEV strategy for inducing change. Because of the state's history and governance arrangements, 
though, linkages between assessment findings and any particular state-level decisions tend to be 
informal. Only one overt link between assessment and the budget is present — the requirement that each 
institution have an approved assessment program in place in order to qualify for addition-to-base 
incentive funds — and as noted, this link has nothing to do with the actual results of assessment. On the 
other hand, there is considerable evidence th.at institutional assessment results are used by SCHEV stai; 
together with other information, to help determine "Funds for Excellence" allocations to institutions or 
in approving new proposed programs of study. Similar connections can be traced between themes 
detected by SCHEV staff across a range of institutional assessment efforts and state-sponsored initiatives 
in faculty development. Finally, SCHEV staff report that assessment provides them with an unmatched 
occasion to discuss curricular issues at the local level. Increased staff understanding gained through this 
process, in turn, has had a considerable impact on SCHEV's own programmatic priorities. 

Linkages to Planning 

The relationship between assessment and U21, SCHEV 's principal recent planning initiative, has been 
similarly indirect. U21 itself notes assessment as one of several vehicles required for curriculum 
reexamination and renewal. Similarly, the U21 update reports presented in 1991 and 1993 both mention 
institution-level assessment as having helped to shape faculty attitudes toward greater curricular 
coherence — especially in general education. Unlike other states, however, there is little visible use in 
Virginia of specific assessment results to draw particular conclusions about program needs and priorities. 



CONCLUSION 

Virgiria's approach to assessment policy has been decisively shaped by its decentralized governance 
context and by a deliberate policy choice to pursue institution-centered assessment as the most 
promising available lever for change. Accountability per se was never intended as a central concern. 
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Instead, accountability was to be achieved as a natural byproduct of positive institutional response. It 
was felt that information for state-level planning and policy would be obtained most effectively by 
identifying any common issues and themes reported by the institutions themselves. For the most part, 
SCHEV's initiative has been successful in attaining these original purposes. Eight years later, there have 
indeed been notable gains in the capacities of most institutions to assess and improve their effectiveness 
in undergraduate instruction. At the same time, information gained through assessment has been useful 
to SCHEV in keeping track of local efforts, and in informing indirectly SCHEV's own planning and 
renewal efforts. 

This achievement, however, has not been without problem. Considerable institutional resistance was 
encountered at the outset, especially on the part of the stated three public research campuses. Many 
institutions also took a long time to get started, yielding very little to report publicly in the near term. 
Moreover, increasingly uncertain fiscal conditions had a major impact on the implementation of a 
program whose major features were determined in a more favorable economic climate (providing an 
additional base budget allocation to institutions to pay for assessment, for instance, was a sound strategy 
to obtain full cooperation, but appeared to legitimize institutional decisions to cut back on assessment 
when fiinds got tight). 

Funding shortfalls and an increasingly unfavorable political climate for higher education, however, 
appear to render Virginia's current strategy increasingly insufficient. Public demands for simple statistics 
on performance are becoming hard to ignore, and providing these may become inevitable as higher 
education argues for an adequate budget. Partly stimulated by the need to respond to poor conditions, 
moreover, SCHEV's own policy agenda has moved visibly toward restructuring instructional priorities 
and delivery. This agenda also demands more systematic and systemwide information. Together, these 
trends appear to be shaping a new approach to assessing performance that requires state-level as well as 
institution-centered results. 
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WISCONSIN 
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OVERVIEW 



W 



isconsin's experience with institutional assessment and accountability illustrates 
strikingly the parallel evolution of each as an element of state policy, The origins of 
institutional assessment in Wisconsin were rooted more strongly in traditional 



accountability concerns than in other states. As in states such as Washington and South Dakota, initial 
proposals for assessment included statewide testing and were stimulated by a need to demonstrate 
political responsiveness. Unlike the experiences of Washington and South Dakota, however — where 
such initiatives were ultimately replaced entirely by institution-centered, improvement-oriented 
approaches — accountability-oriented activities never entirely disappeared in Wisconsin. An institutional 
assessment agenda proceeded, actively led and promoted by the university system administration. 
Experiments with statewide testing and with the use of standard statistical indicators continued as well. 

Unlike the situation in other states, moreover, fiscal concerns were strongly present in Wisconsin 
throughout the period in which assessment initiatives evolved. As a result, when fiscal conditions forced 
the strong reemergence of accountability concerns nationally in the 1990s, the state was positioned to 
have it both ways. 



CHRONOLOGY AND CONTEXT 
System Design and Evolution 

Since 1974, the Board of Regents of the University of Wisconsin (UW) has served as the system's 
statutory governing authority. The board has direct responsibility for the state's unified university 
system, consisting of 2 doctoral institutions, 11 four-year campuses, 13 lower-division university 
centers, and university extension. Wisconsin's 16 vocational -technical institutions are governed 
separately by the Board of Vocational, Technical, and Adult Education, which maintains close liaison 
with the UW Board of Regents on statewide planning issues. Twenty-one independent higher education 
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institutions make up the balance of higher education in the state, but they enroll only a small proportion 
of its students. These governing arrangements have been in place since 1974, and they have produced 
a stable and fairly traditional array of systemwide policies and procedures. Since about the mid-1980s, 
the board has become more proactive with respect to master planning for the UW System. 

Student Demographics 

The state's higher education enterprise enrolls about 225,000 full-time-equivalent (FTE) students, more 
than 85 percent of whom attend public institutions. Fields of study approximate national patterns at both 
the baccalaureate and masters/first professional levels, with the exception of agriculture where the 
proportion of graduates is significantly above average. The percent of minority students is lower than the 
national average, reflecting the state's population. 

During the 1980s, one significant difference between Wisconsin and other states was the low levels of 
available funding per student, especially in the system's four-year public sector. Efforts to raise public 
funding levels for higher education in recent years have been intimately bound up with greater 
accountability. More recently, the funding gap has all but disappeared due largely to enrollment 
management policies begun in 1987. 

Chronology of Related Priorities and Initiatives 

Wisconsin's initiatives with respect to assessment and accountability are diverse find have often been 
embedded in other policy initiatives. Nearly all have been conducted under the auspices of the UW 
Board of Regents, though several were stimulated directly or indirectly by actors outside the higher 
education community. Key dates are as follows: 

1986: The board issues Planning for the Future as the state's master planning document. The report 
notes "sliding levels of financial support" for higher education — together with growing enrollment 
demand — as a significant threat to quality. "Assessment of quality and educational outcomes" is one of 
20 recommendations presented for future planning. 

1987: In February, Governor Tommy Thompson writes President Kenneth Shaw of the UW System to 
express "concern about the lack of our ability to measure student progress." The letter supports board 
plans for assessment and the university's plan to implement a systemwide assessment program by the 
fall of 1991. In September, a UW System Assessment and Testing Advisory Council is appointed by the 
UW System president. 
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1988: In January, the UW System sponsors an initial statewide conference on assessment that includes 
representatives from several states with past experience in implementing assessment. A prominent theme 
of the conference is combining what has worked in other states with assessment approaches adopted by 
individual UW institutions. At the same time, the system administration funds several campuses to 
experiment with local assessment. 

1989: In May, the recommendations of the UW System Testing and Assessment Council, first presented 
in September 1988, are accepted as UW System guidelines for assessment. These establish an 
institution-centered assessment approach, much like that initiated in other states. Institutions are required 
to develop their own local assessment programs consistent with a set of systemwide guidelines and to 
report results to the board by 1991. 

1990: The board adopts a comprehensive Academic Quality Program (AQP) that includes the 
establishment of institution-centered assessment programs and a statewide assessment of verbal and 
quantitative skills. 

1991: In February, the board mandates a pilot study of standardized assessment of 
sophomore-level verbal and quantitative skills, using the ACT-CAAP examination. 

1992: In July, the board issues a new report on undergraduate education entitled The 
Undergraduate Imperative: Building on Excellence. The report is the result of an evaluation of the 
"comprehensive state of undergraduate education in the system 0 begun in the spring of 1991, involving 
public hearings and campus visits by a board working group. The report proposes initiatives in four 
areas — general education, improving teaching, helping students to succeed, and improving partnerships 
with K-12 and two-year institutions — and contains the assertion, "for each policy area . . . appropriate 
accountability measures have been built ... to ensure timely action and identification of improvement 
needs, " Consistent with this philosophy, a set of standard statistical indicators (such as graduation rates) 
are preserved as "consumer information" to be part of the regular 1992-93 system profile provided to 
prospective students and the public. 

In August, the Governors Commission on UW System Compensation also issues its report. This com- 
mission was formed in response to ongoing fiscal problems in the system, in particular its inability to 
recruit and retain new faculty. The final report goes well beyond the commission's initial concep- 
tion by raising comprehensive funding and accountability issues. In essence, it proposes to offer 
increased budgetary flexibility for the university system in return for visible accountability measures. 
Among its recommendations is the establishment of a statewide task force charged "to identify . . . 
specific measures of performance" for the university system. 
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In September, the UW System reports to the board on the results of the pilot study using ACT-C AAR 
The board mandates that a similar systemwide study be carried out again once a value-added component 
is available. 

1993: In March, the Governor's Task Force on UW Accountability Measures is established under 
Executive Order 177. Its report, completed by June, recommends the development of core indicators to 
be produced for all institutions as well as specific indicators to be developed by each institution 
consistent with its mission. 



ASSESSMENT OF INSTITUTIONAL EFFECTIVENESS 
State Strategy and Policy Evolution 

As noted, Wisconsin's policy approach was in many ways typical of states that developed 
assessment initiatives in the late 1980s. Approaching the topic somewhat later than others, the board of 
regents was able to learn from other states' experiences. At the same time, at least in the early stages, it 
had the luxury of time: more than three years elapsed between the point at which assessment firs? 
surfaced in Planning for the Future and the formal adoption of an institutional assessment policy. 

The result was an unusually deliberate path of development and a resulting policy that in many ways 
incorporated the best practice of late 1980s assessment policy. I^ie systemwide conference on 
assessment held in January of 1988, for instance, included presentations by nationally-known assessment 
commentators, pioneering institutions like Alverno and Northeast Missouri State, and representatives of 
states with established assessment programs such as Virginia and New Jersey. The message conveyed 
(and warmly received by institutional audiences) was that institution-centered, improvement-oriented 
policies would pay far more dividends for Wisconsin than would mandated testing. Similarly, the UW 
System Testing and Advisory Council produced a document that cited virtually the entire emerging 
literature on institutional assessment practice at that time. Its recommendations, in turn, were highly 
consistent with institution-centered principles advanced by leading contemporary assessment advocates. 
During this time, system administration funded a number of pilot demonstration projects at $25,000 
each, with the requirement that they report their experiences for the benefit of the entire system. 

By the time this deliberate path of development was completed in 1990, however, 
institution-centered assessment had already become part of a wider policy agenda. Broader concerns 
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about undergraduate quality had stimulated the AQP program, and the assessment of institutional 
effectiveness was made a part of it. While this was happening, the North Central Association (NCA) 
issued its own requirements for the assessment of learning outcomes — requirements that mirrored the 
UW System administration requirements and gave validity to the institution-centered, multiple measures 
programs being developed at the institutions. At the same4ime, the Governor's Commission on 
Compensation was beginning its work, the results of which were to reestablish the accountability agenda 
on a new foundation. 

It is important to note that not all UW institutions proceeded at the same pace. Some UW System 
institutions had assessment programs before they were mandated (UW-Stout, UW-Superior), in some 
cases far in advance of the others (UW-Green Bay). Other institutions developed strong programs 
consistent with their missions and began using the results. Many of these were noted favorably in reports 
to the board on the progress of AQP in the period 1990-93. But other campuses did relatively little. By 
September 1992 when the board of regents was asked to identify the next steps, system administrators 
recommended continuing local assessment measures. The board agreed, but also elected to include 
systemwide sophomore testing using the ACT-CAAP once a value-added component was available. 

Process Required 

According to the AQP adopted by the board of regents in 1990, each institution in the UW System has 
been required since fall 1991 to develop a process for assessing students* verbal and quantitative skills at 
the end of the sophomore year. Rather than mandating a single approach, however, the policy allows 
each campus to develop its own measures. In addition, institutions are required to go beyond these 
identified skill areas to assess teaching and learning in general education and in the major — a . 
requirement reinforced by NCAs reaffirmation requirements. Each institution's assessment program is 
expected to be informed by the set of principles embodied in the final report of the UW System Testing 
and Assessment Advisory Council. As noted, these were consistent with the tenets of the most successful 
assessment programs in other states. The principles were as follows: 

• assessment should "be developed with faculty and staff leadership, and student parti cipation" 

• each institution should establish a body to oversee assessment "consistent with local 
governance procedures'* 

• institutions should "survey existing assessment procedures and use them as a foundation for 
further assessment'* 
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• assessment procedures should be "carefully examined to ensure a high degree of validity and 
reliability" 

• "special care should be taken to insure that assessment is fair to ethnic and minority groups, 
both sexes, persons with disabilities, and members of all social classes" 

• "care should be taken to avoid the misuse of assessment data" (examples of misuse 
included the use of results to bar individual student progress or to sanction individual faculty 
members) 

• assessment programs should be "periodically evaluated and revised as necessary" 

In addition, the advisory committee recommended that special funding be provided to support local 
assessment efforts consistent with these principles, and that such support should come from "new 
allocations rather than [from] reallocating existing resources." 

Assessment Domains 

The council's recommendations specified that all institutions develop programs to assess verbal and 
quantitative skills. However, the council also recommended that institutions go beyond these domains to 
"evaluate the educational process in a variety of ways." Among those specifically noted in the text of the 
recommendation were the following: 

• general education (methods to be decided by institutions) 

• proficiency in the major (through such techniques as "comprehensive examinations, projects, 
performances, or portfolios") 

• student development and the institutional environment (including "participation in extracurricular 
activities, attendance at cultural events, use of the library, and the degree of student satisfaction with 
faculty advising, student services, and campus living conditions") 

• the success of graduates (including alumni surveys, employer surveys, and performance and 
placement in graduate programs) 
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Later documents also emphasized the link between local provisions of the AQP and NCA 
assessment guidelines. These required institutions to develop explicit learning outcome goals and to 
periodically assess the attainment of each. The choice of these goals was left up to each institution. 



STATE-LEVEL INDICATORS 



State Strategy and Policy Evolution 

Unlike many other states, Wisconsin included the concept of developing a limited set of state-level 
indicators of effectiveness largely for accountability purposes right from the start. Both the regents' 1986 
recommendation in Planning for the Future and Governor Thompson's 1987 letter to President Shaw 
carried the clear implication that summative measures of performance should eventually be implemented 
and their results should be publicly reported. While local assessment policies were being developed by 
institutions between 1987 and 1990, statewide testing was never really abandoned. 

Pilot studies of the feasibility of statewide testing using the ACT-CAAP were conducted in 1991-92 to 
provide a policy alternative to local assessment. At the same time, the system began experimenting 
with additional publicly-reported standard statistical indicators with the inclusion of descriptive data for 
each institution in the widely-distributed UW System Statistical Profile. But the major impetus for 
this line of development came from an unlikely source: the Governor's Commission on System 
Compensation. 

Wisconsin differed from other states in the 1980s because fiscal difficulty was a major theme for the 
state. By 1985, what had once been a "model system" was significantly underinvested, a fact noted 
prominently in the regents' 1986 report. The Commission on Compensation was convened in response to 
a specific manifestation of this problem, the growing inability of the UW System to recruit and retain 
nationally-prominent faculty. Addressing this problem, however, required the commission to raise 
additional issues. 

Foremost among them was the fact that existing regulations and practices often prevented the 
system's institutions from using available resources appropriately and flexibly. After considering these 
problems, the commission recommended an alternative, more decentralized approach. But they made 
its implementation conditional on the system's adoption of an explicit set of accountability 
measures. Specifically, the commission recommended that "the System should be held accountable in 
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[the areas of] effectiveness, efficiency, quality, access, diversity, stewardship of assets, and contribution 
to compelling state needs," and that an additional task force be appointed to "identify the specific 
indicators to be utilized to measure performance in these general areas." 

The resulting Governor's Task Force on UW System Accountability Measures completed its report in 
June 1993, after only three months of work. Its findings strongly endorsed the notion of budgetary 
flexibility being contingent on accountability. Both reports also made prominent use of such terms as 
stakeholders and continuous quality improvement in advocating the adoption of performance measures. 

Recognizing this shift in language, both the commission and the task force acknowledged that core 
governance and management practices would eventually be affected. In his transmittal memo to the 
governor, for instance, the chair of the task force emphasized that "the proposed accountability system 
redefines the relationships and responsibilities among UW System officials, legislative and executive 

branch decisionmakers, and the citizens of Wisconsin We believe that it recasts these relationships 

in ways that are healthier and more productive " Consistent with this statement, the task force 

recommended the development of performance indicators around three priority areas: (1) delivering a 
high-quality undergraduate education, (2) meeting the needs of business and other constituency 
organizations, and (3) being "customer-oriented and responsive to customer concerns." And in contrast 
to prior assessment policies, they additionally recommended that reports should be public, that explicit 
standards or targets of performance be adopted, and that there "be consequences for failing to act to 
meet the accountability goals and rewards for special efforts which lead to success in meeting the 
goals." 

Superficially similar to the legislative report-card approaches of states such as Kentucky and South 
Carolina, these recommendations represented a fundamentally different strategy. First, the perspective of 
reporting was placed clearly upon documenting the benefits of its higher education system to the state 
and its citizens. Second, the potential linkage between results and policy action was made clear: 
indicators were intended not only to report on performance after the fact but also to guide the 
development of overall system policy. Similarly, system officials welcomed the opportunity to make 
accountability measures specific and predictable; in the past, they felt they had frequently been 
"whipsawed" by disconnected and often contradictory definitions of performance. 

Finally, the intent of reporting was not just to document system performance but also to positively 
shape institutional behavior. A clear objective, embodied in the third recommendation of the task force, 
for instance, was the development of a systemwide commitment to accountability as reflected in 
changed institutional management practices, greater consciousness of student needs and opinions, and 
the development of informative consumer information. 
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Specific Domains Reported 



As a result of this history, systemwide indicators in Wisconsin are of two kinds. The first is 
composed of statistics that are readily available from the system's unit record enrollment files and 
contained in the UW System Statistical Profile report, issued since 1992-93. This profile primarily 
contains descriptive information such as enrollments, numbers of faculty, and degree programs offered. 
But specific elements of institutional performance also are included in the document: 

• numbers of degrees granted (undergraduate and graduate) 

• percentage of undergraduates completing programs (both at the institution and at any other UW 
campus) 

• average time to completion for undergraduates 

• percent of undergraduate courses taught respectively by regular faculty, other instructional staff, and 
teaching assistants 

• average class size 

• percentage of undergraduate lecture sections enrolling more than 100 students 

The second reporting component consists of the indicators proposed by the Task Force on Accountability 
Measures, which followed the lead of the Commission on Compensation. These are far more extensive 
than the data elements contained in the Statistical Profile and, in many cases, will require new kinds of 
data collection. Suggested indicators are grouped under the headings proposed in the report of the 
Commission on Compensation: 

Quality 

• results of student satisfaction surveys on such matters as course availability, accessibility of faculty, 
advising, and "other aspects of the undergraduate experience*' 

• results of alumni surveys on such matters as career status, further education, and civic participation 

• the percentage of lower-division undergraduate enrollments and contact hours taught respectively 
by regular faculty, other instructional staff, and teaching assistants 

• the dollar value of externally funded research and contracts 
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Effectiveness 

• results of the systemwide competency test (ACT-CAAP) or alternative equivalent testing at the 
sophomore level 

• graduation rates for undergraduates (both full-and part-time) 

• GRE and LSAT scores for graduates, job placement rates, professional and graduate school 
acceptances, pass-rates on licensure examinations, and "other data showing the post-graduate 
experience of UW undergraduates" ■ 

Efficiency 

• average credits-to-degree for baccalaureate graduates, and the proportion of starting freshmen 
who complete in four years, by residency 

• proportion of payrolled employees and funding for institutional support., academic support, 
instruction, and student services 

Access 

• percentage of qualified resident undergraduates accepted for the system as a whole, and the 
proportion of Wisconsin high school graduates enrolling immediately in the UW system each 
year 

Diversity 

• progress toward meeting established affirmative action and equal opportunity goals for staff and 
faculty 

• number and residency of minority students, together with their graduation rates 

• reported number of incidents of sexual harassment 
Stewardship of Assets 

• recruitment and retention rates for faculty, and resources dedicated to faculty development 

• progress on addressing preventive maintenance needs for university facilities 

• number and severity of accidents, injuries, etc., incurred by the UW system 
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Contribution to Compelling State Needs 

results of surveys of employers on such matters as the career preparation of graduates and 
responsiveness of the university system to their needs 

• enrollments in continuing education programs and the assessment of students participating in 
such programs 

About half of the indicators proposed by the task force are already collected in some form by the 
system, but many are not compiled or calculated as indicated. All but one of the remainder (pass-rates 
on licensure examinations) are items that the system has already committed to collecting. 



REVIEWING AND REPORTING OF RESULTS 
Comparative Data Use 

With regard to comparative data, the Wisconsin approach is to try to satisfy two sets of needs. With few 
exceptions, each institution is required to collect standard statistical indicators such as those already 
contained in the profile and recommended by the task force. At the same time, institutions are free, and 
in fact encouraged, to develop and report their own, mission-specific indicators to be placed alongside 
core comparative statistics. Obviously, if the full intent of the task force recommendations were to be 
implemented, comparative data use would be mandatory, because standards would be set and 
consequences attached to performance at designated levels. At the same time, in both the profile and the 
task force recommendations, the state has shown unusual care in ensuring that such statistics are 
reported fairly and interpreted correctly. Explicit safeguards in reporting, for instance, are provided when 
numbers are small or statistically suspect or when the presentation of a given statistic alone might lead 
to misinterpretation. 

Use of Quantitative and Qualitative Data 

Virtually all indicators currently used in the profile and proposed by the task force are quantitative. 
Some of the latter, however, are less numeric than they are indicative of the presence or absence of 
progress on a given dimension — for example, progress in meeting affirmative action goals or in 
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addressing preventive maintenance needs for facilities. The text of the task force report also emphasizes 
that indicators can be either quantitative or qualitative, although the majority of the examples actually 
provided are of the former. 

Use of Contextual Data 

The use of data is at present unknown, although the grouping of proposed task force indicators by 
analytical category will presumably require some explanation. The profile suggests that such a strategy 
will likely be followed. Although quantitative results in the profile are presented in tabular form to 
enable ready comparisons among institutions, each table is also accompanied by a brief explanation that 
cautions readers about making inappropriate comparisons. At the same time, the profile itself is 
contained in a larger descriptive context, whiclv{5resents the mission and important characteristics of 
each campus. 



CONNECTION TO OTHER PROCESSES 

Although there is a clear intent to do so, accountability and assessment have as yet not been explicitly 
linked to other state-level decision processes. Institution-level assessment practices consistent with 
system policy are intended to be locally run and applied; the board has not as yet received detailed 
reports of institutional activities and results upon which policy judgments, if intended, could be based. 
Statewide statistical indicators are also too new to allow them to serve as a reasonable guide to policy. 

Linkages to Budget 

Wisconsin's current strategy with respect to accountability indicators is, in at least one respect, strongly 
tied to budget. The existence and effective operation of the proposed accountability system is seen by 
the board as a key to leveraging budget flexibility. Faced with the proposed contingency contained in 
the compensation commission's proposal for new kinds of accountability measures, the board has been 
willing to accept these as a precondition for a new budget compact. In this sense, Wisconsin's linkage 
between performance indicators and budgeting resembies that of Tennessee. For both states, the 
symbolic and politiqal importance of the initiative for higher education's leadership far outweighs the 
direct value of the indicators themselves in informing policy. But implicit in the report of the 
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Accountability Task Force are some long-term implications. The recommendation that explicit 
standards be established and that consequences for institutions be associated with these standards clearly 
signals an intent to go farther. It is unclear at this point the degree to which the board intends to use 
indicators to establish new systemwide priorities based on what is found (as, for example, is beginning 
to occur in states such as Virginia). 



Linkages to Planning 

As in the case of resource allocation, it appears that the system intends eventually to use the resulting 
data to help inform system-level planning. Because the initiative is so new, however, this premise has 
yet to be demonstrated. Unlike such states as Virginia or Tennessee, current initiatives in assessment and 
accountability in Wisconsin did not arise from a deliberate planning context in which past assessment 
results are presented to the public, together with plans and recommendations for the coming years. 
Planning documents in Wisconsin do address assessment as policy — but as yet contain little assessment 
data. 



IMPLEMENTATION PROBLEMS AND CONCLUSION 



The overwhelming conclusion of the Wisconsin experience is that it is vitally important to keep options 
open. Rather than abandoning a systemwide indicator approach when many other states did so in the 
mid-1980s, Wisconsin embraced an institution-centered approach, even as it continued to experiment 
with the ACT-CAAP and to refine available performance measures derived from unit record data. In 
this, the UW Board of Regents had substantial advantage over their counterparts in such states as 
Virginia and Texas. But in doing so, Wisconsin was also well positioned to distinguish clearly between 
institution-based improvement-oriented assessment at the campus level and necessary statewide 
accountability measures. This distinction appears particularly important in today's state policy context. 
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EFFECTIVENESS IN UNDERGRADUATE EDUCATION: 
AN ANALYSIS OF STATE QUALITY INDICATORS 



Richard C. Richardson, Jr. 

Higher education in the United States is an enterprise of considerable magnitude. In the most 
recent year for which data are available, public colleges and universities spent more than $85 
_billion to educate nearly 11 million students. Not surprisingly, policymakers want to know 
more about the returns on this higher education investment. Efforts to define suitable indicators for 
assessing quality have become a priority in many states. If state coordinating and governing boards fail 
to satisfy the needs of elected officials, legislators may mandate measures that fail to take into account 
legitimate institutional concerns. 

A working consensus is needed about the nature of quality and what purposes quality indicators should 
serve. This paper attempts to answer four fundamental questions: 

1. What indicators have been developed among the 10 states that appear to be in the vanguard of 
efforts to assess the outputs and outcomes of their higher education systems? 

2. How has the development of particular indicators been affected by changes in higher education 
governance? 

3. How can indicators be designed to reconcile the tensions between a professional emphasis on 
institutional improvement and a state policy emphasis on institutional accountability? 

/ 

4. What are the implications of assessment and accountability measures for state and institutional 
policy? 

This analysis begins by considering briefly how states form and implement higher education policies. 
The experiences of the 10 states are then examined to see how specific state-level indicators have been 
used. The indicators are considered from the planning perspective and in the context of efforts to define 
quality and "reinvent government." The analysis leads to the conclusion that state and institutional 
assessment goals may be more compatible than many believe. Finally, a conceptual framework is 
presented for identifying and defining indicators that may help reduce tensions between efforts 
aimed at institutional improvement and those focused on accountability. 
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POLICY FORMATION AND IMPLEMENTATION 



Higher education policies reflect the core social values of choice, equity, efficiency, and quality. 1 Choice 
involves the freedom to pursue (or not pursue) one's own preferences in one's own way. Equity relates 
to who attends which institutions and studies which majors, and what happens to them as a consequence. 
Efficiency concerns the benefits of education in relation to the costs. Useful indicators must 
acknowledge the close relationships between quality and the other three sometimes complementary, 
sometimes conflicting values. 

Each state selects quality indicators within a political and fiscal environment produced by its own 
distinctive history and culture. The use of quality indicators makes the most sense in a state where 
higher education purposes and priorities have been explicitly identified and communicated to 
institutions, and where the same or similar priorities have been pursued for some period of time across 
governors, legislators, and other policymakers. The rule is always, "What is valued gets measured, and 
what is measured gets valued." 

Appointed and elected state officials use four types of instruments to encourage institutions to respond 
to state policy. Mandates prescribe actions, inducements provide monetary incentives for desired actions, 
capacity-building helps to develop human and physical resources essential to desired actions, and 
system-changing creates new authority for responding to public needs. 2 To avoid institutional 
resistance, inducements and capacity-building are the preferred policy instruments under ordinary 
circumstances. Among the case-study states, Virginia uses challenge grants to make quaiity enhancement 
a priority. They have also been prominent in New Jersey and Ohio. However, special programs often 
produce limited results as they lack leverage for changing mainstream practices (readjusting the 
balance between research and teaching, for example). Also, unlike appropriated base funds, special 
add-ons are highly vulnerable to funding cut-backs, as shown by the reduction or outright elimination of 
challenge programs in all three states mentioned above. 

State leaders often find that indicators help them make tough choices between competing values and 
priorities. Appointed officials, who typically experience closer institutional ties, often prefer 
indicators that provide insight into improving institutional performance, although it is rare for these 
officials to use the indicators as summative judgments to reward or punish institutions (the Tennessee 
Performance Funding Program is an exception, of course). Elected officials, who often have less 
technical knowledge, less time, and more skepticism about institutional intentions, are more inclined to 
use the quality indicators to punish or reward institutions; they tend to believe that assured compliance is 
better than inducements offered in the hope some institutions will choose to respond. Both perspectives 
can be found in the 10 case-study states. 



132 



DEVELOPMENT AND USE OF INDICATORS IN CASE-STUDY STATES 



Linkages to Planning and Budget 

Some states have used quality indicators in conjunction with a strategic planning process in order to 
measure system or institutional progress toward achieving state priorities. Tennessee, for example, has a 
significant strategic planning history. Its early use of performance funding was designed to ensure 
institutional attention to state priorities. The master plan for 1987-93 3 sets forth 5 principles and 15 
targets. The principles establish goals related to mission definition; general education competencies; 
relationships with other sectors of education, business, and industry; quality; and access. The targets 
represent measurable objectives to be attained during the period covered by the plan. Some targets 
clearly require state-level policy action (e.g., "develop comprehensive financial aid programs and 
reward meritorious institutional performance"). Other targets call for action at the institutional level 
(e.g., "increase the level of job placement for two-year graduates," or "exceed the national norm in bac- 
calaureate student performance"). 

While the approach taken by Tennessee has been driven by state higher education priorities, higher 
education policymakers have been more successful in "developing and reporting state-level indicators 
and in using then effectively to account for and promote the state's higher education system than 
they have been in inducing institutions to engage in their own local planning and assessment 
exercises." 4 Perhaps this is the case because "performance data have been used to make a general 
budget case" with relatively little evidence of the information being used analytically to "identify 
particular areas of future-funding needs." 5 

Colorado, Florida, Kentucky, and South Carolina all appear to have been influenced by Tennessee in 
their adoption of indicators. The difference is that accountability was imposed by legislators rather than 
by a state coordinating board. In none of the four states was much consideration given to ongoing 
planning in the choice of indicators. However, state political leaders did consult with coordinating 
boards in Kentucky and South Carolina in the implementation of the selected indicators. In Colorado 
and Florida, institutions were expected to assist in implementation. In all of these states, imposing 
indicators provoked institutional anxieties and some resistance. Currently, each state requires a set of 
indicators to be reported annually in a format useful to policymakers. 

The planning process itself has been the focur of efforts to develop and report quality indicators in 
Illinois, New York, Virginia, and Wisconsin. Illinois is a particularly strong case-study example. In this 
state, assessment measures have been the product of group planning focused on defined areas (e.g., 
undergraduate education, productivity, faculty, and affordabiiity). With the exception of Illinois, where 
indicators have been used to identify programs that are candidates for merger or discontinuance, 
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there is little evidence of institutional anxiety or resistance to assessment in these four states. In part, this 
is because the process has been very evolutionary and incremental. Also, New York, Virginia, and 
Wisconsin are still in the very early stages of development, with no apparent agreement at this point on 
common institutional indicators that might permit interinstitutional comparisons. The approach taken by 
these four states is consistent with professional and institutional concerns about a focus on improvement. 
At the time the case studies were undertaken, however, none of these states had been successful in 
producing the type of report sought by state policymakers, one that would allow for institution-level 
examination and comparison. 

The Texas approach is the best example of a state combining accountability with institutional 
improvement. Ironically, it is the state that has arguably done the least amount of higher education 
planning. This changed in 1991 with the adoption of HB 2009, which mandated strategic plans for all 
state government agencies, including university systems and higher education institutions: "Plans, which 
must be updated every two years, are designed in part 'to provide a context to link the budget and other 
legislative processes to priority issues, to impose continuity in budgeting, and to improve accountability 
for the use of state resources'." 6 The legislation also required regular performance audits. 

Like Te> Jiessee, Texas intends to link indicators to performance funding. The 1991 General 
Appropriations Act provided for performance state aid to be distributed by the coordinating board only 
after certifying that institutions had attained appropriate performance levels on eight indicators 
established by the legislature. While no funds were appropriated for performance state aid in 1991, the 
coordinating board has been working on funding formulas that will take these indicators into account. 



CATEGORIZING QUALITY INDICATORS 
Inputs/Outputs/Outcomes Model 

The model Texas has employed is a useful way of classifying indicators. The state planning process 
distinguishes among "statements of purpose, statements of direction and statements of impact." 
Statements of purpose relate to agency mission. Statements of direction are concerned with agency goals, 
objectives, strategies, and action plans. Central to the development of these are inputs defined as "the 
resources that an agency uses to produce services, including human, financial, facility or material 
resources." 7 Statements of impact give rise to output and outcome measures. Output measures "are tools 
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or indicators, to count the services and goods produced by an agency." Outcome measures "are tools or 
indicators, to assess the actual impact of an agency's actions." 8 

Texas is not alone in its recognition of the need to tie priorities to the budget. A popular book among 
state policy officials, Reinventing Government: How the Entrepreneurial Spirit is Transforming the 
Public Sector, identifies the budget as the lever that drives government most powerfully and argues that 
results-oriented organizations must find ways to "develop budget systems that fund outcomes rather than 
inputs." The authors add that one of the simplest means of accomplishing this is to add output and/or 
outcome measures to a "mission driven budget." Outcomes are a measure of quality; outputs are a 
measure of the volume of what an organization produces. 9 

Using the definitions supplied in the Texas strategic planning guide and in Reinventing Government, it is 
possible to classify the indicators identified in the 10 case-study states. Table 1 lists input indicators. 
None of the 15 indicators in the input category has been identified by more than 4 states. All of the 
case-study states that adopted input indicators appear to have done so to improve quality, efficiency, or 
equity. Most states are not very interested in input measures unless they have some obvious utility such 
as measuring progress toward achieving a state priority. 

Table 2 summarizes state activity in relation to output indicators. Clearly, states have more interest in 
outputs than inputs. In fact, the number of input measures included among state indicators would 
probably be even fewer if it were not for institutional pressures to include measures that demonstrate 
underfunding in comparison with peers in other states. There also is greater consensus among states 
about essential output indicators. All the states are concerned about enrollment, retention, progression, 
and graduation data. Most monitor for several categories of student as evidence of equity as well as 
efficiency. State concerns for equity and efficiency also are apparent in the attention given to students in 
remedial courses. State priorities for economic development are reflected in the reporting of external or 
sponsored research funds. Other state priorities reflected among the output indicators include articulation 
between two- and four-year institutions and the avoidance of unnecessary program duplication. Obvious 
state interest in output measures reflects concern about the relationship between costs and benefits in the 
services provided by public agencies. 
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An emerging state interest in outcome measures is evident in table 3. While some of the indicators 
identified as output measures in table 2 might be classified as outcome measures, the defining 
characteristic of outcome indicators is their effort to identify qualitative differences in institutional 
performance. Often this is done, as in the case of admission standards and measures, by comparing 
institutional intentions with performance. Outcome indicators consider differences in institutional 
mission and objective to a greater degree than either input or output measures. Because of their 
complexity, good outcome measures are difficult to define. Kentucky, South Carolina, Tennessee, and 
Wisconsin are clear leaders in using these types of indicator. Illinois, New York, and Virginia are among 
the states demonstrating the most interest in their potential use. Interestingly, all are strong planning 
states. 

The chronological development of input, output, and outcome measures reflects the stages through 
which state coordination of higher education has passed. Prior to the early 1950s, individual institutions 
interacted directly with legislatures in all 10 of the case-study states. At that time higher education 
represented a small part of most state budgets, and input indicators repiesented a reasonable 
approximation of quality understood as resource availability. 

In the post World War II period, the growth of the higher education enterprise raised concerns about 
efficiency and access and led to the establishment of coordinating boards. To the earlier input measures 
of resource-based quality, coordinating boards added output measures, since efficiency can only be 
assessed by examining outputs in relation to inputs. 10 

By 1984, state concerns began to focus on quality. The convergence of economic pressure and renewed 
attention to quality issues on the part of American business produced an uneasiness among state 
policymakers. New models aimed at reporting qualitative outcomes, such as the New Mexico Report 
Card, found ready acceptance. 

Quality Definition Model 

Current interest in quality improvement has produced a number of sophisticated definitions that afford 
opportunities for integrating state and institutional perspectives. The following five definitions are useful 
in assessing quality in state settings. 11 

1 . Reputation or institutional ranking (transcendent quality) Those who attend institutions with the 
highest status are perceived to have attained better educations than the graduates of lesser 
institutions. This definition speaks more directly to resources than to performance. Transcendent 
quality includes the input indicators identified earlier in table 1. 
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2. Relationship between inputs and outputs (cost/benefit quality) Table 4 provides a listing of 
indicators that reflect costftenefit quality. While most of these measures are used in state funding 
formulas, they rarely are considered by state policymakers as a measure of quality when defined as 
inputs. However, comparing and contrasting input and output indicators, especially when compared 
with system and institutional goals, provide valuable measures of quality and efficiency. 



TABLE 4 

INDICATORS OF COST/BENEFIT QUALITY 


COST/BENEFIT QUALITY INDICATORS 


STATES 


Total degrees awarded by institution and discipline 


FL, IL, KY, TN, WI 


Total student credit hours produced by institution and discipline 


FL, IL, KY, NY 


Student transfers between two- and four-year institutions 


IL, NY, SC, TN 


1 Total contact hours of instruction by faculty rank, institution and 
J course level (faculty workload) 


FL, KY, NY 


| State appropriations per resident student 


CO, NY, VA 


1 State appropriations per capita 


CO, NY, VA 


Classroom and lab utilization 


FL, IL, KY 


Sustained financial commitment to instruction 


CO, NY. WI 


J Average class size 


IL, VA, WI 


j Time to degree and number of credits by institution and degree 


FL. KY, WI 


| Average faculty salary 


CO,IL 


Alumni and private contributions 


CO, NY 


Student characteristics 


IL, SC 


Charges to students 


IL, NY 


Two-year transfer students completing four- year degrees 


IL, KY 


Student/faculty ratios 


CO 


Facilities maintenance 


WI 
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3. Consistency of indicators matched with stated standards (process-based quality) The most 
common application of this definition occurs in accreditation. Indicators derived through the 
accreditation process are part of an institution's regular assessment practices and, therefore, 
require little oversight and have no added cost. Indicators derived from the accreditation process 
also help to guard against inappropriate comparisons that fail to consider differences in missions 
and resources. Table 5 provides a summary of the process-based quality indicators in use among 
the case-study states. 



TABLE 5 

INDICATORS OF PROCESS-BASED QUALITY 


PROCESS-BASED QUALITY INDICATORS 


STATES 


Admission standards and measures of the first-year class 
against standards 


CO, IL, NY, TN, VA 


Number and percentage of accredited programs and 
programs eligible for accreditation 


IL, KY, SC, TN 


Availability of academic programs 


CO, EL, NY 


Faculty diversity 


CO, NY, WI 


Student course demand analysis 


FL, KY 


Availability of student financial aid 


CO 


Upper division undergraduate students participating in 
sponsored research programs 


SC 


Students who have some kind of summarizing experience 
(thesis, recital, comprehensive exam) in major 


VA 


Undergraduate students who have a small class or 
seminar experience 


VA 


Reporting & resolution of sexual harassment complaints 


WI 


Faculty retention & development 


WI 


Workplace safety 
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4. Measurable attributes of graduates (product-based quality) Examples of this include per- 
formance on licensing exams or measures of general education competencies. One difficulty with 
interpreting tlies^ indicators is separating the effects attributable to the program from those caused 
by differences in entering-student characteristics. For instance, in one state pass-rates on a nursing 
licensing exam for graduates of some community colleges regularly exceed rates for 
baccalaureate graduates of four- year institutions. In part this is because there are large numbers of 
students enrolled at the community colleges who already hold baccalaureate degrees or who have 
substantial prior college experience. The most useful product-based measures, like their 
cost/benefit counterparts, pay attention to appropriate input measures. The product-based quality 
measures summarized for the case-study states in table 6 typically do not associate output measures 
with their corresponding inputs. 



TABLE 6 

INDICATORS OF PRODUCT-BASED QUALITY 


PRODUCT-BASED QUALITY INDICATORS 


STATES 


Enrollment, progression, retention and graduation 

by race, gender, disability and high school achievement 


CO, FL, IL, KY, NY, SC. TN, 
TX, VA, WI 


Pass-rates on professional licensure exams 


CO, FL, IL, KY, NY, SC. TN, 
TX, WI / 


External or sponsored research funds 


CO, IL, KY, NY, TX, VA, WI 


Students in remedial courses; students exiting 
remedial courses and completing entry-level courses. 


KY, SC. TN, TX, VA 


Lower division courses taught by full-time faculty, part-time 
faculty and graduate assistants 


SC. TN, VA, WI 


Placement of graduates 


SC. TN. VA , WI 


Student assessment results 


IL. NY, VA, WI 


Student performance on nationally normed exams 


IL. NY, WI 


Continuing education/extension enrollments and 
public service activities 


IL, KY, WI 


Course completion rates 


NY, TX 


National faculty and student awards 


NY 
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5. Client satisfaction (user-based quality) Users in this case are not only alumni, students, and 
parents but also the business and professional community. User-based quality is at the core of 
the Total Quality Management movement. As indicated in table 7, the typical measure of 
user-based quality is satisfaction as reported on some type of survey. 



TABLE 7 

INDICATORS OF USER-BASED QUALITY 


USER-BASED QUALITY INDICATORS 


STATES 


Student, alumni, parent, client, and employer satisfaction 


FL, KY, NY, WI 



Because this study focuses on state policy arrangements, the popularity of client satisfaction-surveys 
among the case-study states is likely understated. Process-based and user-based measures of quality 
are favored by institutions because of their compatibility with marketing strategies and with 
regional accreditation practices. These forms of quality indicators are less appealing to elected 
officials who prefer the more quantitatively oriented cost/benefit and product-based measures. 



CHARACTERISTICS OF USEFUL INDICATORS 

The categories presented in this paper are only two of several schemes for conceptualizing quality. 8 
The input, output, and outcome terminology used by state policymakers is compatible with 
Alexander Astin's approach to assessment, 12 but it goes further by linking assessment to emerging 
state and institutional interests in assessment as a prerequisite to efficiency and improvement. D. T. 
Seymour 13 and Cornesky et. al. 14 offer helpful perspectives, but their approach is too tightly wedded 
to the institutional perspective to maintain satisfactory comfort levels among state policy officials. 
The advantage of the approaches presented here rests in the way indicators are used in planning or 
are linked to definitions of quality that drive current institutional practice. The two conceptual 
arrangements also help to clarify the basis for disagreements among state and institutional 
representatives on the appropriate indicators for assessing system and institutional performance. 
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Useful indicators of educational quality for those who form and those who implement policy must 
address at least three interrelated questions: 



Where do quality indicators fit conceptually among the strategies used to define and 
implement state priorities? How will the results of assessment be used? 

Conceptually, quality indicators can be seen as part of a strategic planning process through which states, 
in cooperation with institutions, define priorities for improvement and keep track of the results. They 
may also be seen as part of an effort to control costs and to assure product quality. By taking a 
planning and improvement perspective, each state can choose unique indicators in terms of its 
priorities at a given time. Emphasizing accountability leads to a greater number of common measures 
across states and over time. 

Realistically, ihere is no way of limiting how quality indicators are used in formulating policy. Certainly 
they will be used to make incremental adjustments when institutional outcomes appear inconsistent 
with policy objectives. However, resisting the development of indicators because they might be misused 
is not a promising strategy. Higher education officials are in a stronger negotiating position when they 
define the indicators that serve implementation purposes and acknowledge the concerns of elected 
officials as well as those of institutional representatives. 

What goals or objectives have been defined as priorities for a state system of higher 
education? For how long have they been pursued? 

Each state defines its own priorities as pan of an ongoing process to allocate the resources needed to 
promote the four core values of quality, equity, efficiency, and choice. States should select indicators that 
estimate progress toward the priorities they pursue over time. In this respect, indicators may be useful in 
identifying aspects of a system that require attention. Because of differences in the way state systems are 
organized, indicators should be collected in aggregate form for the system as well as for the institutions 
that comprise the system. 

What criteria credibly measure progress toward the attainment of state goals and 
priorities? From whose perspective should they be assessed? 

Different actors define quality in ways that emphasize their individual purposes and perspectives. 
Institutional actors emphasize process-based and user-based indicators. Governors and legislators are 
oriented more toward cost/benefit and product-based measures. As Seymour has pointed out, "higher 
education no longer has the luxury of using narrow definitions of quality/' States need a variety of 
indicators to help them shape responsive systems of higher education. 
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CONCLUSION 



Quality indicators appear most useful if integrated in a planning process designed to coordinate 
institutional efforts to attain state priorities. As this report has shown, case-study states have 
conceptualized their indicators in one of two approaches. In the first (summarized in tables 1-3), the 
indicators have been reported according to an input, output, and outcome typology. In the second 
(summarized in tables 4-7), the same indicators have been rearranged into cost/benefit, 
process-based, product-based, and user-based categories to help make the point that different users 
prefer different measures. 

The continuum from strategic planning to accountability/improvement seems to capture the range of 
current state activity. Texas and Illinois are examples of the strategic planning approach. Wisconsin 
(which uses such categories as quality, effectiveness, efficiency, access, diversity, and stewardship of 
assets) and Colorado (which uses excellence, access and diversity, efficiency, and adequacy of 
resources) are examples of the accountability/improvement approach. States considering the adoption of 
quality indicators for monitoring a system of higher education would be well-advised to incorporate 
requirements for planning, institutional improvement, and accountability in their design, since all three 
must be addressed over time. 
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DEVELOPING STATEWIDE PERFORMANCE INDICATORS FOR 
HIGHER EDUCATION: POLICY THEMES AND VARIATIONS 

Peter T. Ewell 

Performance indicators as tools for state higher education policy have many roots. Some can be 
traced directly to public higher education's new accountability-oriented climate, itself the 
product of increasingly harsh fiscal realities and growing public concerns about higher 
education costs. Some can be traced to the earlier development of state-based assessment efforts in the 
mid-1980s, intended to improve higher education's effectiveness by stimulating institutions to examine 
their own outcomes and operations. Others can be attributed to long-standing efforts on the pan of a 
number of states to develop comprehensive planning systems for guiding the overall development of 
public higher education. 

Whatever their origins, state-based higher education performance indicators are clearly a 
phenomenon. 1 Approximately one-third of states now have such indicators in place, the vast majority 
of which have been developed and implemented within the last three years. Based on the experiences 
of 10 case-study states, the purpose of this analysis is to examine this phenomenon descriptively, to 
attempt some early generalizations about its evolution and future potential, and to draw some 
conclusions about its implementation. 

The first section of this analysis provides some general background for the development of 
performance indicators, including a brief review of earlier state-based assessment efforts and their 
growing limitations in the face of new policy demands. The second section provides an overall 
description of policy implementation in the 10 states, including an analysis of themes and experiences 
common to many. The third presents a systematic review of the various types of indicator currently 
being collected or proposed. The fourth and final section attempts some deeper analytical conclusions 
and suggests some broad lessons that can be drawn for other states. Hazarding generalizations from a 
rich variety of case-study material is always a risky business, necessary though it may be. Readers are 
strongly encouraged to review the case studies themselves and to draw their own conclusions. 

BACKGROUND 

The development of state-based performance indicators in higher education must be viewed against the 
backdrop of a decade-long process of policy evolution. Beginning in the early 1980s, state policy 
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increasingly began to emphasize themes of return on investment as well as the linkage between higher 
education and the attainment of wider public goals such as work force preparation and the development 
of an educated citizenry. 2 In many states, academic concern about revitalizing the undergraduate 
curriculum coincided with policy interest in improving what was seen as an important but neglected 
aspect of higher education's responsibility. 3 

The policies that resulted emphasized the assessment of education outcomes but also involved the 
creation of a range of other mechanisms for inducing quality improvement in undergraduate education, 
often in the form of addition-to-base, incentive-funding mechanisms. 4 These policy tools were well 
suited to the state policy climate of the times, emphasizing curricular renewal and experimentation in the 
context of ample resources. The state-mandated assessment programs that emerged during this period 
strongly resembled one another in their desire to fuse in a single process the dual agendas of improving 
instruction at the institution level and demonstrating accountability on a statewide basis. At the same 
time, they tended to be implemented as add-ons to existing policy, often separately funded and rarely 
directly integrated into state-level planning and budgeting mechanisms. 5 

Such approaches generally allowed institutions maximum discretion in determining what and how to 
assess; at the same time, they assiduously avoided interinstitutional comparisons. Most required 
institutions to engage in assessment within broadly defined domains and to report their findings publicly. 
State-level review of reported results was viewed as much a means to providing institutions with 
feedback on their evolving assessment programs as it was a mechanism for ensuring compliance. 
Indeed, the policy objective of such efforts was consistent throughout: to create an ongoing, 
locally-owned process of continuous self-examination and improvement. 

Led by states such as Virginia and Colorado, some two-thirds of the states had such institution 
centered programs in place by 1990. 6 But this approach always represented something of a gamble for 
state higher education leaders. If institutions could indeed field credible assessment methods quickly and 
could report actions taken as a result, then accountability demands could be met in large measure. But 
this decentralized improvement-oriented approach depended decisively on two conditions. 7 First, 
institutions had to implement credible local assessment programs promptly. By 1990, it was all too clear 
that institutional response was both lagging and uneven. Second, institutions had to find an effective 
way to communicate the results of assessment to outside audiences. Institution-centered reports were 
rich but diffuse, although they rarely allowed policymakers the opportunity to compare institutions or to 
examine overall system performance. 

These drawbacks became increasingly salient in the harsh policy conditions of the early nineties. New 
fiscal realities meant that neither states nor institutions could continue to afford assessment as an add-on, 
and many simply ceased investing in the process. Simultaneously, tight times meant the reemergence of 
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traditional public concerns about higher education's efficiency, embracing issues such as faculty 
workloads, program duplication, and investments in undergraduate teaching. This was brought to a head 
when funding shortfalls demanded that public officials make increasingly hard decisions about where 
and how to invest. 

For these reasons, the conversation accompanying the recent emergence of performance indicators 
differs considerably from its counterpart around assessment some 10 years ago. For one thing, current 
debate has a considerably wider domain. While the emergence of state interest in assessment in the early 
1980s signalled a new concern with quality as opposed to more traditional issues of access and 
efficiency, current initiatives embrace both. At the same time, a new Equality rhetoric" is apparent in the 
labels attached to many of these initiatives — Priorities, Quality and Productivity (PQP) in Illinois or 
Academic Quality Program (AQP) in Wisconsin, for instance — a vernacular that strongly reflects the 
growing influence of Total Quality Management concepts drawn from business and industry. 

The actual array of statistical performance indicators emerging in many states thus mixes old and new 
elements, but in a conscious attempt to be comprehensive. Many efforts appear dominated by 
concerns about traditionally defined efficiency, but equally many — for example, South Carolina's 
requirement to report the number of undergraduate students participating in sponsored-research 
activities, or Virginia's and Wisconsin's proposals to examine the proportion of students engaging in 
particular types of auricular or classroom activities — reflect an emerging concern with explicit 
educational processes and delivery. 

The inclusion of such domains suggests a new willingness to examine the actual production function of 
higher education. If this trend continues, and if it is more deliberately linked to planning and resource 
allocation as many states suggest, the result may be an unprecedented level of policy engagement in 
managing instructional delivery. 



IMPLEMENTING INDICATORS 



While many of the obstacles and issues faced by indicators resemble those encountered by earlier 
attempts to deploy assessment or program review initiatives, there are considerable differences. 8 Among 
the case-study states, for example, a number of themes have emerged that were not present in the earlier 
initiatives: 
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Quick implementation with little prior conceptual development 

The majority of indicator proposals deployed in the case-study states arose extremely rapidly and rely 
heavily on existing information or readily available data. As a result, few have been guided by prior 
analysis of statistics that would identify those most appropriate for systemic decision making and public 
reporting. In many cases, rapid implementation was a direct consequence of the legislative source of the 
initiative. Kentucky's SB 109 and South Carolina's Act 255 (as well as New Mexico's earlier report-card 
bill, upon which both were based) came quickly, primarily because the legislators responsible were in 
direct contact with one another — a pattern appropriately labeled "legislation by FAX." In other cases 
— Colorado's Scorecard initiative and Virginia's and New York's recent indicator proposals, for exam- 
ple — the initiative was developed rapidly by the coordinating or governing agency as a preemp- 
tive strike to anticipate and avoid expected legislative mandates. 

In still others states, indicators came through the back door as a needed auxiliary to make other, more 
important initiatives work as expected. Texas' abortive performance-based aid program and its 
state-agency strategic-planning initiative both required performance data to operate. But initial 
systematic thinking about the needed indicators was largely overwhelmed by the sheer scale of these 
proposals and the consequent press of political events. Similarly, strengthened accountability measures 
in Wisconsin first arose as a necessary counterbalance offered by the UW system in return for greater 
budget flexibility. In neither case was time and opportunity available to develop indicators from 
conceptually sound first principles. 

Illinois and Tennessee provide interesting exceptions to this pattern of breakneck implementation. 
Though different in many ways, initiatives evolved slowly in both states. The ancestry of the statistics 
and analytical procedures used in Illinois* PQP initiative, for example, can be traced back through a 
complex array of earlier program costing, planning, and evaluation efforts that stretch back to the 
mid-1980s. Similarly in Tennessee, the now annual Tennessee Challenge 2000 report was first mandated 
by the legislature in 1989, and that state's higher education commission has taken the opportunity in 
each successive year to refine both the measures used and their manner of presentation. The fact that the 
state had already rehearsed such a program five years earlier in the form of legislatively-mandated 
performance measures also enhanced this base of experience. 
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A focus on interinstitutional comparison 

The majority of institution-centered assessment requirements initiated by states in the mid-1980s 
assiduously avoided comparing institutional performance. With the primary policy focus placed on 

150 



institutional improvement, the development of comparative performance statistics was seen as 
inappropriate at that time, and each institution was encouraged to develop approaches suited to its own 
unique mission and student clientele. But an equally compelling reason for this course of action was 
political. In virtually every state, institutions strongly resisted direct statistical comparisons, and states 
came to view the diversity of measures as a reasonable price to get the program started. In the 1990s, in 
stark contrast, the majority of current performance indicator initiatives are deliberately comparative. 
This is particularly so for those legislatively mandated as in South Carolina, Kentucky, and Texas. But it 
is equally the case for preemptive initiatives such as those in Colorado, Virginia, New York, and 
Wisconsin, states in which the higher education agency has up to now consciously avoided comparing 
institutions. 

Where agencies have been allowed some discretion in setting policy regarding institutional 
comparison, they have also made a visible attempt to ensure that it is appropriate. In Colorado and 
Tennessee, for instance, only sectoral comparisons are provided (although in the case of Tennessee, the 
legislature recently extended the sectoral statistics to include reporting on each institution). In addition, 
the vast majority of initiatives distinguish two- from four-year institutions and often require different 
performance indicators for each. Several states also attempt to buffer the effects of direct comparison by- 
providing additional textual commentary on the reported indicators and how they should (or should not) 
be interpreted. Tennessee Challenge 2000 and the Wisconsin's Consumer Profile provide excellent 
examples of such public reporting. Both include explicit caveats on interpretation where statistically 
risky samples are reported or where knowledge of specific population characteristics is important for 
drawing a valid conclusion. Last year, along similar lines, South Carolina combined the comparative 
statistical reporting required by Act 255 with additional institutional commentary and data unique to 
each institution, generated as a result of the state's earlier institutional effectiveness mandate. Though 
too early to tell, this seems also the route planned by agency leaders in Virginia and New York. 

Given considerable levels of institutional resistance to even the slightest suggestion of comparison, its 
strong presence in current policy initiatives is significant. Clearly, this shift is partly due to increasing 
pressures for familiar, easy-to-read public accountability reports. But comparative reporting has deeper 
consequences, as it also provides a potential means for states to make high-stakes decisions affecting 
institutions themselves. 



Linkages to funding and resource allocation 

Another prominent feature of the 1980s institution-centered assessment mandates was then- 
deliberate avoidance of a linkage to funding. Though assessment policies in states like Colorado, 
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Virginia, and South Carolina included strong budgetary sanctions for institutions that failed to comply, 
the actual results of assessment were not tied to funding. The prominent exception, of course, was 
Tennessee where, through performance funding, assessment results directly determined a portion of the 
allocation. 

By contrast, a number of initiatives in case-study states now contain such a linkage. Perhaps the most 
prominent is Texas, where the legislature in 1992 considered performance-based aid for higher 
education with initial allocation levels set at 10 percent of the base. The expected implementation 
difficulties, together with strong political pressure, put this initiative on indefinite hold, but it illustrates 
the type of once-unthinkable proposal that is now quite fashionable in state policy circles. In other 
case-study states, the rhetoric of indicator development strongly suggests an eventual link to funding. In 
Wisconsin, for example, the 1993 report of the Task Force on Systemwide Accountability explicitly 
recommends that such a mechanism be developed; Kentucky's governor similarly stated last year that 
institutional performance as measured and reported through SB 109 "should ultimately be rewarded 
through our funding system." 

In the majority of case-study states, however, linkages to resources are indirect. Illinois* PQP 
initiative, for instance, has significant budgetary consequences for institutions, though its primary unit of 
analysis is academic programs rather than budget lines. In Colorado, parallel legislative initiatives 
include HB 1110, which requires a legislative set-aside to fund areas of special concern to the state. In 
states such as Virginia and New York, where higher education agencies have largely shaped the 
initiative, such linkages are generally more muted. Planning documents in both states suggest that 
indicators will be used to help identify areas of potential state investment, but they propose no direct ties 
between results and allocation. Indeed, based on its long experience with institution-centered 
assessment, Virginia agency leaders worry about potential distortions in data-reporting if institutions 
begin to directly associate high stakes outcomes with statistical performance. 

K-12 connection 

Early observers of 1980s state-based assessment programs noted the fact that pressures for higher 
education accountability often arose in the language of K-12, an idiom highly familiar to legislators and 
others outside higher education, 9 In most cases, higher education leaders were able to persuade / 
legislators and board members that the profound differences between the two sectors necessitated 
distinctive approaches. As indicator initiatives evolved in many of the case-study states, however, a 
parallel dynamic occurred, and, this time, many of the programs retained a strong K-12 flavor. External 
authorities came to the discussion of indicators with strengthened perceptions that K-12 reform was 
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working and an accompanying conviction that all of public education should be integrated within a far 
more proactive accountability and improvement framework. 

Probably the strongest manifestation of this K-12 connection is in Kentucky, where radical and highly 
visible K-12 reform metres launched in 1990 have had an acknowledged influence on mandated 
higher education porting. In other states, the connection is present but less obvious. In South Carolina, 
the linkage is long-term: the original 1988 Cutting Edge legislation that resulted in 
institution-centered assessment also mandated comprehensive K-12 reform; Act 255 was seen by many 
as bringing the two systems more fully in line with respect to accountability. Similarly in Tennessee, 
Tennessee Challenge 2000 was launched as both a K-12 and a higher education initiative, and, since 
1989, its associated reporting has required a joint report by the Tennessee Higher Education Commission 
(THEC) and the state's department of education. Significantly, however, the visible K-12 connection in 
case-study states has largely been confined to legislative initiatives. In states such as Virginia, New 
York, Illinois, and Wisconsin, where higher education agencies have been the major actors, this linkage 
is much less apparent. 

Not taking "no" for an answer 

In contrast to earlier assessment conversations, state authorities in case-study states have been far less 
willing this time around to be talked out of the specifics of proposed requirements when faced with 
initial (and often vociferous) protests from institutions. While institution-centered assessment mandates 
in the mid- to late- 1980s in states such as Virginia, Colorado, and South Carolina involved considerable 
institutional input and were often substantially modified as a result, indicator proposals in most 
case-study states have been more strongly backed by state policymakers and, partly as a result, have 
been more often accepted by institutions as inevitable. 

Probably the best illustration of this dynamic is in South Carolina, Here Cutting Edge legislation first 
raised the issue of effectiveness reporting in 1988, In developing the required initiative, however, the 
higher education agency worked closely with a specially created institutional advisory committee (the 
SCHEA Network) in negotiating a workable set of arrangements. When Act 255 passed four years later, 
institutions were surprised to find little sympathy for their input in the legislature; as one observer 
colorfully put it, "they were cut off at the knees/* Much of the same pattern was apparent in Kentucky 
with SB 109, although here the initial institution-centered alternative put forward by the coordinating 
board was not as far along as its South Carolina counterpart when the legislative mandate came down. 
Similarly in Colorado, where the initial highly intrusive requirements of HB1 187 were gradually 
converted by the coordinating board into a workable institution-centered system, development of the 
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preemptive Scorecard involved far less institutional input. 10 Also in Colorado — and in states like 
Wisconsin and Illinois — state policymakers had the substantial advantage of having already collected 
the data required, meaning that institutional approval v/as less needed. 

Finally, institutions have been less ready to push the issue than they were five years ago. Many appear to 
have accepted stronger accountability measures as inevitable, or at least as the necessary price for 
retaining continued support and discretion in decision making. Others are far more preoccupied with 
responding to major funding shortfalls than with arguing about additional reporting. This is certainly not 
to say that institutions have been silent, particularly in states such as Illinois where initiatives like PQP 
involve real allocation decisions. But, in many ways, this exception illustrates the larger point: 
institutions prefer onerous statistical reporting to much more intrusive policy alternatives. 



INDICATORS OF WHAT? 



In the majority of case-study states, performance indicator initiatives contain many of the same 
elements despite their diverse origins. Most include some 15 to 20 distinct data items collected by the 
governing or coordinating board and reported in tabular form to facilitate public communication and to 
allow comparisons among institutions. This convergence in content is due more to circumstances than 
consensus: because only a small body of available state-level data is available and since there are limited 
technical possibilities for manipulating it, indicator initiatives often end up alike. But some of this 
congruence results from shared perceptions about the importance of particular pieces of information. 

Indicators currently proposed or in use in the 10 case-study states are presented in table 1. Items 
included under instructional inputs are for the most part familiar and already collected. In Florida and 
Texas, statewide testing mandates cover this ground, while other states include admissions standards and 
additional measures of institutional selectivity. Reflecting earlier concerns about student preparation, a 
number of states require reporting on remedial activities and their effectiveness. 

Items included under instructional processes and use of resources are a mixed bag of old and new. Many 
are traditional activity measures such as the distribution of credits by discipline and level. A new and 
suggestive theme is gauging faculty workload by breaking down instructional activity indicators by level 
or type of faculty associated with them. This reflects obvious concern with ensuring student access to 
senior faculty, especially in the lower division. Six of 10 case-study states contain such a measure 
explicitly, with an additional state suggesting interest. A similar though more diffuse concern is evident 
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with small classes (Wisconsin and Virginia), and specific types of instructional experiences (e.g., 
seminars in Virginia, or student participation in sponsored research in South Carolina). Finally, 
traditional efficiency concerns are apparent in a new guise through a focus on persistence and 
time-to-degree issues; indicators of this type are included in 6 of 10 states. 

Items presented under instructional outcomes reflect continuing interest in this topic. Most frequently 
noted are graduation/completion rates, the only type of indicator included by all 10 case-study states. 
Also evident are a variety of post-graduation statistics, including pass-rates on licensure examinations, 
job placement, and graduate satisfaction with instruction. Interestingly, few new cognitive measures 
appear to be currently on the table. ACT-COMP scores and major-field testing results in Tennessee, as 
well as CLAST scores in Florida, have been in place for many years; the only exception here is the 
proposed sophomore testing in Wisconsin using the ACT-CAAP examination. 

The majority of items noted under the heading of efficiency and productivity, and condition of the asset 
are very traditional. The former consist primarily of standard efficiency and cost measures that have for 
the most part been collected by these states for over a decade. The latter contain familiar funding 
ratios and measures of externally sponsored research activities, as well as newer items that demonstrate 
growing concern about the ways institutions have reacted incrementally to recent funding shortfalls. 

Items included under diversity/access/equity reflect both ongoing state-level concern with these topics 
and the ready availability of state-level data. Eight of ten states, for instance, report graduation and/or 
persistence statistics by race and gender, and almost as many explicitly monitor enrollment activity by 
ethnic group. Others broaden the diversity issue to include faculty statistics and affirmative action 
activities. But relatively few states address the question of access for all students. Only two, for instance, 
propose indicators related to overall participation rates or costs to attend. 

Items included under articulatiort/K-12 linkages reflect particular concern that institutions of higher 
education pay more attention to the educational system as a whole. By far the most common measure is 
volume and performance of two-year college transfers at senior institutions. In three of five states 
including such indicators — Florida, Tennessee, and Texas — these were in place and reported 
previously. Explicit concerns about K-12 linkages are less directly apparent; only two states, for 
instance, mandate feedback to high schools. 

The final heading, relation to state needs, reflects an emerging interest in examining higher 
education outputs and capacity in relation to state priorities. Only three states suggest such indicators at 
this point: Illinois, New York, and Wisconsin. Interestingly, these are also states in which the higher 
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TABLE 1 : Analysis of State-Level Indicators in the Case-Study States 





INSTRUCTIONAL INPUTS 


INSTRUCTIONAL PROCESSES/ 
USE OF RESOURCES 


INSTRUCTIONAL OUTCOMES 


COLORADO 


o ACT/SAT scores of entering freshmen 


o Availability of academic programs 
o Sustained financial commitment 

to instruction 
o Student/faculty ratios 


o Graduation rates by ethnicity 
o Performance of graduates on 
licensure exams 


FLORIDA 


o Mandated basic skills testing 


o Total credits produced by discipline 

o Total contact hours by faculty rank/level 

o Time to degree and number of credts required 

o Course demand analysis 

o Classroom utilization 


o CLAST examination on basic skills 
o Total degrees awarded by discipline 
o Performance of graduates on 

licensure exams 
o Alumni/employer follow-up responses 
o Retention/graduation rates by ethnicity 
o Job placement rates for 

vocational students 


ILUNOIS 


o Student demand (PQP) 


o CentraJity of programs to mission (PQP) 
o Faculty worWoad 
o Time to degree 


o Persistence and graduation rates 

o Follow-up of graduates 

o Success of graduates (PQP) 


KENTUCKY 


o Number of students in remediation 


o Total credits produced by discipline 
o Total contact hours by faculty rank/level 
o Specific faculty workload measures 
o Time to degree and number of credits 

to complete 
o Course demand analysis 
o Classroom utilization 


o Total degrees awarded by discipline 
o Graduate performance on 

licensure exams 
o Graduate/employer satisfaction 
o Persistence and graduation rates 

by ethnicity 


NEW YORK 


o Trends in admissions data, high school 
graduates, etc. 


o Time to degree completion 

o Freshman-to-sophomore persistence 

o Percentage resources for undergraduate 

instruction 
o Class size by level 


o Graduation/persistence rates 
o Graduate performance on licensing 
exams 

o Student perceptions of quality 


S. CAROUNA 


o Number and performance of remedial 
students 


o Lower-division courses taught by faculty type 
o Upper-division students in sponsored research 
o Graduate students drawn from in-state 


o Graduation/completion rates 

o Placement of graduates 

o Grad, performance on licensina exams 


TENNESSEE 


o Remedial students and effectiveness 

of remediation 
o ACT scores of entering students 


o Graduation/completion rates 

o Lower-division courses taught by faculty type 


o Grad. performance on licensing exams 

o ACT-COMP scores 

o Major field test scores 

o Student satisfaction ratings 

o Job placement rates for araduates 


TEXAS 


o TASP examination results 


o Wrthin-term persistence rates 
o First-year retention rates 
o Classroom utilization 


o Graduation/completion within 6 years 
o Grad. performance on licensing exams 


VIRGINIA 


o Admissions standards and requirements 
o Number of students meeting admissions 

standards 
o High school courses taken 
o Levels of remediation required 


o Average class sizes 

o Percent undergraduates taught by 

senior faculty 
o Small ciassy seminar experiences 
o "CaDStoneVinteqrative expenences 


o Graduation/completion rates 
o Employment/graduate school 
placement rates 


WISCONSIN 


o ACT/SAT Scores 

o Resident Students Accepted * 

o W1 high school graduates enrolled * 


o Average time to completion 

o Courses taught by faculty type/level 

o Average class size 

o Percent classes of 1 00 or more 


o Numbers of degrees granted 
o GraduatiorVcompletion rates 
o Graduate satisfaction/outcomes * 
oACT-CAAP Results* 



• Indicators from the Governor's Task Force on UW Accountability Measures. 
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BEST COPY AVAILABLE 



EFFICIENCY/ 
PRODUCTIVITY 



CONDITION OF THE 
ASSET 



DlVERStTY/ACCESS/EQUITY 



ARTICULATION/ 
K-1 2 LINKAGES 



RELATION TO 
STATE NEEDS 



o Student/faculty o Total revenues per student 
ratios o Alumni and private contributions 
o State approp. per resident student 
o State approp. per capita 
o Grant and contract dollars per FTE 
o Average faculty salary 



o Graduation rates 

by ethnicity 
o Availability of student 

financial aid 
o Faculty diversity 
o College participation rate 



o Retention/graduation rates 
by ethnicity 



o Performance of transfers at 
senior institution 



o Program costs (POP) 
o Faculty workload 



o Number/proportion 
of accredited programs 



o Various measures in 
individual reports 



o Persistence and graduation 
rates by ethnicity 



o Feedback on performance to 

high schools 
o Performance of transfers at 

senior institution 



o Relation of program to 
employment needs (POP) 



o Research and service in 

support cf K-1 2 
o Number and performance of 

two-year transfers 



o Credits per faculty 

produced 
o Student/faculty ratios 



o Graduate program and faculty 

"quality" measures 
o Condition of campus facilities 
o External fundraising success 



o Number/proportion of 
programs accredted 



o Number/proportion of 
programs accredited 



o Student demographics 
o Persistence/graduation 

by ethnicity 
o Ethnicity of faculty 
o Trends in "costs to attend" 



o Graduates in science, 

engineering, etc. 
o Economic impact on state 



o Number and trends in 
minority enrollment 



o Transfer rates and transfer 
performance 



o Minority enrollments 
o Co mp tetion/g raduation rates 
for minorities 



o Two-year college transfer rates 
and performance 



o Value of externally-sponsored 
research 



o Minority first-year persistence o Mandated feedback on peformanoe 
o Minority performance to high schools 

00 TASP o Mandated feedback on performance 
to two-vear colleges 



o Quality of institutional 
assessment program 

o Level of sponsored 
research activity 



o Sponsored resarch 
levels * 



o Faculty recruitment/retention * 
o Faculty development resources * 
o Maintenance investments * 
o Accidents/injuries * 



o Graduation/completion rates 
by ethinicity 



o Progress in affirmative action * 
o Minority graduation rates * 
o Sexual harrassment incidents * 



o Employer ratings of 
"responsiveness" * 

o Continuing education 
activities * 

157 



9 

lERJC 



1« 



education agency has led the initiative and has proceeded in the context of a wider strategic-planning 
framework. 

The overall weight of attention suggested by this array of indicators — despite the intent to be 
comprehensive — suggests a particular concern with instructional processes and outcomes at the 
undergraduate level. While not surprising, this conclusion is significant when coupled with the insistence 
with which such measures have been advanced by state authorities and with their common comparative 
focus and potential linkages to funding. In this sense, the center of gravity of state-level policy intent 
appears quite consistent with earlier undergraduate reform initiatives of the eighties including, quite 
prominently, assessment. 



POLICY PATTERNS AND CONCLUSIONS 



Table 2 provides an overview of the 10 case-study states with respect to the development of 
accountability and assessment policy. For each state, separate lines are shown for predominant initiatives 
in "institutional effectiveness" and in state indicator reporting. For Florida and Texas, the other line is 
included to address unique statewide student testing policies. Each initiative is then briefly described 
under a number of headings. The first three identify the initiative in terms of who, when, and what. 
Where a joint listing occurs in the who initiated column, the initiative was a collaborative effort among 
the parties noted. The strategy column briefly analyzes the intent of these initiatives, while the two 
linkage columns rate the degree to which each is tied to budget and to statewide planning efforts, 
respectively. 

Inspection of the array of strategies across case-study states reveals an interesting variety. For 
institutional effectiveness initiatives, the improvement strategy suggests a primary intent to stimulate 
institutions to create their own local change processes by mandating an assessment process and publicly 
reporting its results. This theme is dominant in 6 of the 10 states. Tennessee's and Texas* reward 
strategy suggests a somewhat different approach, based principally upon the creation of concrete 
budgetary incentives for change. Exceptions to these common categories are Illinois* comprehensive 
approach that relies upon a range of state and institutional studies and inducements, and Florida's 
catch-up, indicating the late development of both institutional effectiveness and state indicator 
initiatives. 
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Table 2: A Comparative Overview of Ten State Initiatives 



Who 

Initiatiated? 



When 
Initiated? Initiative 



"Strategy 1 ' 



Link to 
Budget 



Link to 
Planning 



COLORADO 

"Institutional Effectiveness" Legislature 

State Indicators SHEEO 
FLORIDA 

"Institutional Effectiveness" Legislature 

State Indicators Legislature 

Other Legislature 
ILLINOIS 

"Institutional Effectiveness" SHEEO 

State Indicators SHEEO 
KENTUCKY 

"Institutional Effectiveness" SHEEO 

State Indicators Legislature 
NEW YORK 

"Institutional Effectiveness" System 

State Indicators System 



SOUTH CAROLINA 

"Institutional Effectiveness" Legislature 

State Indicators Legislature 



1985 HB1187 
1991 "Score Card" 



Improvement Sanction 
"Pre-emptive" Low 



1991 PEPC Study "Catch-up" Unknown 

1991 "Accountability Report" "Catch-up" Unknown 



1979 CLAST 

1986 Undeigrad study 

1991 PQP 

1990 KAEP 

1992 SB109 

1987 Assessment 

1993 "Indicators" 

1988 "Cutting Edge" 
1992 Act 255 



Gatekeeping Ixw 



Low 
Low 

Unknown 
Unknown 
Low 



Comprehensive Moderate High 

"Realignment" Moderate Moderate 

Improvement Low Low 

"Standards" Low/planned Low 



Improvement 

"Budget 
Bargain" 



None 
Low 



Low 

Low/planned 



Improvement Sanctions Low 
"Standards" Low Low 



"Institutional Effectiveness" SHEEO 



State Indicators 



TEXAS 

"Institutional Effectiveness" 



State Indicators 
Other 



VIRGINIA 

"Institutional Effectiveness" 
State Indicators 
WISCONSIN 

"Institutional Effectiveness" 
State Indicators 



Legislature/ 
SHEEO 



Legislature 

SHEEO/ 
Legislature 



SHEEO 
SHEEO 

Gov/SHEEO 

Blue Ribbon/ 
SHEEO 



1979 Performance 
Funding 

1989 TN2000 



1992 Performance Aid 
1987 TASP 

1986 Assessment 

1993 [Indicators] 



1987 AQP 

1992 System 

Accountability 



Rewards High 



"Standards" Low 



"Budget 
Bargain" 



Moderate 



Moderate 



Rewards High Low/planned 

Gatekeeping Moderate Low 

Improvement Sanctions Moderate 

"Pre-emptive" Unknown Unknown 



Improvement Low Low 

Low/planned Low/planned 
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For state indicator initiatives, the preemptive strategy suggests an attempt by the state agency to take 
the lead in developing policy to head off or anticipate legislative action. This strategy is 
predominantly present in Virginia and New York — both states in which higher education authorities 
have been granted substantial latitude to develop policy, but where there is growing accountability 
pressure. The standards strategy suggests an attempt to use publicly-reported statistical indicators as a 
direct lever for institutional change, much as these states have previously used achievement standards in 
K-12. States exhibiting this strategy are all southeastern (Kentucky, South Carolina, and Tennessee), and 
the majority of initiatives are legislatively mandated. 

The budget-bargain strategy, in turn, suggests a policy intent to trade additional accountability 
measures for more resources or for increased budget flexibility. This strategy is strongly apparent in 
New York and Wisconsin, both large and centralized statewide systems attempting to purchase greater 
autonomy. Arguably, Tennessee could be put in this category as well, as Tennessee Challenge 2000 is 
seen by state higher education agency leaders as an important budget leveraging device. 

Finally, Illinois' realignment strategy once more appears unique: PQP involves the complex use of 
indicator data, together with other information, as part of a more comprehensive attempt to restructure 
the state's program inventory and to align instructional-delivery processes to meet new environmental 
conditions. 

The relations of these initiatives to budgeting and statewide planning efforts are similarly varied. Most 
linkages to resource allocation are low and indirect, with some interesting exceptions. Sanctions are 
used when an institution's failure to report data required under institutional effectiveness policies has 
potential budget consequences, a feature present in Colorado, South Carolina, and Virginia. In Texas and 
Tennessee, of course, performance funding yields strong budgetary ties (note that in Texas, TASP results 
are already used to help allocate remedial dollars to institutions). Illinois' initiatives similarly rate a 
moderate classification because of the anticipated effects of program inventory reduction and 
realignment efforts. 

Most linkages to statewide planning appear similarly modest at this point Tennessee and Virginia show 
greater-than-average connections, but largely because assessment results are included in statewide 
planning documents for reporting. There is little evidence to date that either state has prominently used 
reported data to help determine statewide priorities or otherwise to shape the actual content of planning 
recommendations. Again, Illinois appears an exception, as indicator initiatives are firmly embedded in a 
wider strategic planning context with a considerable history. Inspection of table 2 also reveals some 
overall clusters of state policy experience with respect to assessment and accountability. The first pattern 
can be appropriately termed the one-two response and is strongly present in Colorado, South Carolina, 
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Kentucky, Virginia, and, to some degree, New York. Under this pattern, institution-centered assessment 
programs were in place and operating by 1990. But escalating pressures for accountability not 
addressed by such an approach resulted in a more recent additional state-indicator approach imposed by 
legislation or launched preemptively by the state higher education agency. 

The second pattern, one shared by Tennessee and Wisconsin, is the parallel strategy. This means that 
both institutional effectiveness and state indicator initiatives have developed over a fairly extensive time 
period, with a predominant state policy focus of trading greater accountability for increased 
budgetary allocation and authority. To some extent. New York also fits this pattern, particularly with 
respect to the implied budget bargain. 

The third pattern, exhibited by Florida and Texas, is the strategy of statewide testing. In both cases the 
very existence of such programs appears to have diverted the kinds of assessment and accountability 
conversations occurring in other states. As a result, when these issues eventually did arise, little had been 
done in either arena. 

Finally, as noted, Illinois appears to have engaged in a policy development strategy. Illinois is 
distinct from other case-study states: the state has explicitly avoided separate development of either an 
institutional effectiveness or an indicator initiative. Instead, it has incorporated aspects of both into a 
more comprehensive set of planning, review, and action initiatives. Certainly, this path of development 
has resulted in a more coherent policy approach than in other states. Whether it has allowed the state's 
higher education system to meet growing demands for accountability in an effective manner remains to 
be seen. 

Review of case-study materials suggests a number of additional policy conclusions that other states may 
find interesting. Among the most salient are the following: 

• The need to recognize a growing divergence between the agendas of improving institutional 
effectiveness and of achieving public accountability 

State assessment mandates of the mid-1980s were typically advanced and implemented under the dual 
banner of accountability and institutional improvement. In designing institution-centered assessment 
initiatives, state officials were aware of the tension between these two objectives, but they were hopeful 
that a positive institutional response (and the ability to demonstrate it to legislative authorities) would 
effectively discharge accountability obligations. For a variety of reasons already noted, this did not 
happen. As a result, the experiences of case-study states have increasingly sustained the proposition 
that local improvement and accountability are different agendas that cannot be adequately 
addressed through a single policy process. 
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The principal evidence for this contention is the dominance of the one-two response pattern noted above, 
a pattern based on the growing inability of institution-centered assessment initiatives to deliver the 
goods on accountability. Interview data drawn from case-study states strongly confirm this 
contention. In Virginia, state officials readily admit the growing need for both kinds of processes. 
Counterparts in Wisconsin emphasize the breakthrough in thinking that occurred when concepts of 
assessment and accountability, previously considered almost automatically interchangeable were 
separated. In Florida, after the Postsecondary Education Planning Commission (PEPC) was able to use a 
late start to examine the experiences of other states, it also proposed a dual-track process involving a 
combination of locally-centered improvement efforts and statewide accountability measures. An agency 
staff member in Kentucky made the overall point well: "You can miss the point and get real fuzzy if you 
try to do everything together." 

• The growing importance of the "budget-bargain" 

More than simply satisfying increasing demands for public reporting, the experiences of the 
case-study states suggest the utility of explicitly trading strengthened accountability measures for 
additional budgetary benefits. Some aspects of this budget-bargain have already been mentioned. In 
Wisconsin, for instance, the entire indicators conversation grew out of the deliberations of the Blue 
Ribbon Commission on University Compensation, one of whose conclusions was that increased 
budgetary autonomy might help university administrators manage more effectively for the future. But 
the commission also believed that such freedom ought not to be granted without additional assurances 
that higher education would act responsibly, and it recommended that additional accountability measures 
be studied as well. New York's experience, as noted, was similar. Here, however, higher education 
officials themselves took the initiative in proposing performance indicators, hoping that this action 
would help smooth the way for legislative approval of far more important proposals about budget 
flexibility within the SUN Y system. 

Tennessee's experience provides an additional illustration of this bargain, but in a different direction. 
Here performance funding has always been viewed by state higher education officials as being at least as 
important in leveraging budget as in inducing institutions to improve delivery. 11 Data generated through 
the process, and additionally through 1984 legislative benchmarking requirements and Tennessee 
Challenge 2000, have been seen as particularly valuable in making higher education's case to the 
legislature, especially when most other competitors for funding can routinely present information about 
results. It is claimed that the evidence for the success of this approach lies in the fact that Tennessee 
higher education has maintained its share of available state dollars in comparison with other agencies 
and functions, a pattern that is rare elsewhere. 



162 



• The importance of a conceptual framework and a link to action 

As noted, the majority of indicator proposals in case-study states were developed quickly, either as 
political initiatives or in anticipation of them. Hence the need to act quickly limited the domain of these 
proposals to information already available or easily collected. In few cases was there the leisure to 
develop indicators from first principles, either in terms of a conceptual framework of important policy 
questions or an explicit analysis of constituency interests and demands. This is not to say that states 
were unaware of the desirability of doing so. Indeed, in many cases, state agencies consciously 
attempted to group indicators into analytical categories or to make sense of what would otherwise have 
been a laundry list. In Colorado, for instance, indicators included in the Scorecard have been organized 
under four common educational values: excellence, access, efficiency, and adequate resources. Similarly, 
Wisconsin's task force recommendations suggested the development of indicators around key areas of 
quality, effectiveness, efficiency, access, diversity, stewardship of assets, and contribution to compelling 
state needs. 

Examining the broader linkage between indicators and planning in case-study states, it is apparent that 
state leaders have conceived of performance measures more often as mechanisms for tracking progress 
rather than as a means to help actively identify areas of need or priorities for planning. One clearly 
expressed motive for the passage of Kentucky's SB 109, for instance, was "to monitor achievement" 
of the state's strategic plan, an attempt also made in South Carolina with a newly developed report 
combining Act 255 and Institutional Effectiveness results. In Virginia, moreover, the primary mechanism 
for public reporting of the results of institutional assessment has been through the biennial Virginia Plan 
for Higher Education, which also contains proposed initiatives for the next biennium. Probably the 
most sophisticated connection is in Tennessee, where indicator statistics mandated through Tennessee 
Challenge 2000 and derived from performance funding are presented annually in the form of a 
planning update that ties statistics to specific planning objectives and an associated budget request. On 
balance, policymakers in each of these states have had more success in using indicators to document 
goal achievement than to help make hard decisions or to plan for the future. 

• The importance of time and of policy consistency 

Another prominent lesson 'rom the experience of case-study states <s the considerable benefit of time. 
Those states with long-siding accountability initiatives or a clear tradition of state planning and 
reporting appear to have evolved far more useful approaches than those lacking these advantages. 
Tennessee and Illinois provide excellent illustrations, though they have engaged in very different policy 
efforts. In Tennessee, the decade-long development of performance funding, combined with a 
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considerable fund of state experience with performance indicators dating back to 1984, gave policy 
leaders and institutions time to get used to operating in a statistical environment and provided ample 
opportunities to evolve sophisticated accountability reports for outside audiences. In Illinois, moreover, 
statewide planning and review efforts by the board of higher education reach back to the mid-1970s and 
have gradually evolved to encompass the kinds of statistical measures enacted de novo in other states. 
The experience of both states demonstrates the utility of a consistent long-term policy agenda of which 
indicators are a part. 

Flexibility and sheer passage of time are also important. Wisconsin's current accountability 
initiatives, for example, began as far back as 1986 with a statewide testing proposal. Rather than 
completely abandon this proposal as other states did at that time, state officials kept their options open 
as they simultaneously pursued more institution-centered alternatives and additional statistical 
indicators. Time and consistency, of course, are luxuries enjoyed only by agencies already allowed 
substantial policy discretion. In South Carolina and Kentucky, and to some extent in Colorado and 
Florida, higher education policy leaders were compelled to react to externally developed proposals on a 
short time frame, precisely because existing political conditions had rendered them largely unable to 
create an adequate conceptual foundation and a more rational policy approach. 

• The importance of an explicit communications strategy 

The experience of case-study states also suggests that meeting growing accountability demands requires 
more than just developing and implementing indicators. State officials must also think through carefully 
some specific mechanisms for communicating performance statistics to particular audiences. 

As noted previously, the most frequent format for communicating results is a general purpose report that 
integrates performance information with other descriptive and contextual material. Prominent examples 
are Tennessee Challenge 2000, South Carolina's new ci ubined Act 2551 Institutional Effectiveness 
report, and the Virginia Plan for Higher Education. But case-study states also exhibit a variety of 
additional reporting mechanisms tailored for specific audiences. In Wisconsin, for example, statistical 
performance indicators on instructional processes and outcomes were first introduced in the form of a 
profile, included with descriptions of the university system campuses in materials distributed to 
prospective students. Similarly, Illinois briefly pursued a set of campus-specific consumer reports as part 
of its comprehensive array of accountability efforts; in this case, rather than being produced 
centrally in a comparative format, individual institutions were free to develop their own documents for 
distribution to local constituents and the wider public. 

Whatever their quality, complex analytical documents based on institution-level data sources are 
unlikely to be sufficient to meet growing accountability demands. Some means will have to be found to 
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communicate the big picture efficiently to unschooled audiences, despite inevitable distortions and 
misinterpretations. 



CONCLUSION 



Underlying all of these conclusions is a single message for state policy leaders: to be effective, 
indicators must be seen as a tool to shape the future, not a collection of statistics to report the past. 
Without a clear framework for interpretation and a wider state strategy for change, statistical indicators 
may help states solve a short-term accountability problem, but they will be unlikely to assist higher 
education policy leaders in accomplishing the increasingly urgent tasks of realignment and 
priority-setting. Though pioneers in many respects, few case-study states have as yet been able to 
accomplish this transition. This has been largely because higher education leadership has been unable, 
on its own, to develop a clear state consensus on policies and priorities. As a result, the question of what 
kinds of data and performance measures are appropriate to guide future development has not been 
systematically addressed. Well beyond the technology of indicators themselves, this is a task requiring 
systematic attention on the part of all state-level stakeholders, not just those few charged with 
responsibility for higher education. 
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